-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DC] Meraki Device Connector #290
Merged
sfc-gh-afedorov
merged 58 commits into
snowflakedb:v1.8.6
from
chanzuckerberg:aku/meraki_connector
Sep 10, 2019
Merged
Changes from all commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
c001f5e
first commit
kuannie1 1d8a45b
meraki connector skeleton (needs custom updating)
kuannie1 b0e4801
rebase
alldoami b30174b
meraki logo
alldoami 24f0664
changed secrets
alldoami 92b01f0
a connector that runs. next: making more tables and double checking v…
kuannie1 179325b
create two tables, has numeric value empty string is not recognized e…
alldoami f509053
removed None
alldoami d777a21
deleted unnecessary functions and included db.py functions
kuannie1 706624a
debug
alldoami 81f4a11
ingesting meraki client & device info and inserting into landing table
kuannie1 327ae3f
Update src/connectors/meraki.py
kuannie1 fd457c3
first commit
kuannie1 fcc2520
meraki connector skeleton (needs custom updating)
kuannie1 2a8c475
rebase
alldoami bb027a5
meraki logo
alldoami d0eb88a
changed secrets
alldoami a881d9b
a connector that runs. next: making more tables and double checking v…
kuannie1 e413652
create two tables, has numeric value empty string is not recognized e…
alldoami d7bb036
removed None
alldoami 47859d9
deleted unnecessary functions and included db.py functions
kuannie1 db03993
debug
alldoami e7487e1
ingesting meraki client & device info and inserting into landing table
kuannie1 b154502
Update src/connectors/meraki.py
kuannie1 d39979e
Merge branch 'aku/meraki_connector' of github.com:chanzuckerberg/Snow…
kuannie1 f78dc8a
made some successful changes but pushing before I break anything else
kuannie1 4bf02cb
combined table variables
alldoami a0f99f4
fixed try except network
alldoami 409b2d0
fixed syntax
alldoami f7d259d
deleting redundant pictures
kuannie1 801d6cc
Merge branch 'aku/meraki_connector' of github.com:chanzuckerberg/Snow…
kuannie1 64276bc
remaining changes
kuannie1 54ef870
changed whitelist input type
alldoami 279338c
inserting clients in batches
kuannie1 fe2fb8f
batching clients
kuannie1 e7be0aa
resolved merge conflicts
kuannie1 f903a47
small changes
kuannie1 a9f25dd
more small changes
kuannie1 dc91f32
default value for get
alldoami eef66e7
fix merge
alldoami 09bffa6
fix get to default to None
alldoami 26d1589
unfinished meraki file
kuannie1 2ff4139
only 1 input table in ingest()
kuannie1 6260259
Revert "unfinished meraki file"
kuannie1 9d8b261
only 1 table in ingest()
kuannie1 af7c6c4
Update src/connectors/__init__.py
kuannie1 a386650
Eduardo's suggestions from code review
kuannie1 c020007
Apply suggestions from code review
kuannie1 75e301e
renaming meraki to meraki_devices and code review suggestions
kuannie1 a8f2f38
andrey's feedback
kuannie1 7ae202a
Delete cisco_umbrella.png
sfc-gh-afedorov a0fb70c
Merge branch 'v1.8.6' into aku/meraki_connector
sfc-gh-afedorov c09881b
Update __init__.py
sfc-gh-afedorov 383b997
Update meraki_devices.py
sfc-gh-afedorov b466ccc
followed pep8 standards
kuannie1 b147f05
Merge branch 'v1.8.6' into aku/meraki_connector
kuannie1 158ccd8
meraki works with WebUI
kuannie1 78acd49
Update meraki_devices.py
sfc-gh-afedorov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,7 +14,7 @@ | |
from datetime import datetime | ||
|
||
|
||
PAGE_SIZE = 50 | ||
PAGE_SIZE = 1000 | ||
|
||
CONNECTION_OPTIONS = [ | ||
{ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,185 @@ | ||
"""Meraki Devices | ||
Collect Meraki Device information using an API Token | ||
""" | ||
|
||
from runners.helpers import log | ||
from runners.helpers import db | ||
from runners.helpers.dbconfig import ROLE as SA_ROLE | ||
|
||
from datetime import datetime | ||
|
||
import requests | ||
from urllib.error import HTTPError | ||
from .utils import yaml_dump | ||
|
||
|
||
PAGE_SIZE = 5 | ||
|
||
CONNECTION_OPTIONS = [ | ||
{ | ||
'name': 'api_token', | ||
'title': "Meraki API Token", | ||
'prompt': "Your Meraki API Token", | ||
'type': 'str', | ||
'secret': True, | ||
'required': True, | ||
}, | ||
{ | ||
'name': 'network_id_whitelist', | ||
'title': "Meraki Network Ids Whitelist", | ||
'prompt': "Whitelist of Network Ids", | ||
'type': 'list', | ||
'secret': False, | ||
'required': True, | ||
}, | ||
] | ||
|
||
LANDING_TABLE_COLUMNS_CLIENT = [ | ||
('insert_id', 'NUMBER IDENTITY START 1 INCREMENT 1'), | ||
('snapshot_at', 'TIMESTAMP_LTZ(9)'), | ||
('raw', 'VARIANT'), | ||
('id', 'VARCHAR(256)'), | ||
('mac', 'VARCHAR(256)'), | ||
('description', 'VARCHAR(256)'), | ||
('mdns_name', 'VARCHAR(256)'), | ||
('dhcp_hostname', 'VARCHAR(256)'), | ||
('ip', 'VARCHAR(256)'), | ||
('switchport', 'VARCHAR(256)'), | ||
('vlan', 'INT'), | ||
('usage_sent', 'INT'), | ||
('usage_recv', 'INT'), | ||
('serial', 'VARCHAR(256)'), | ||
] | ||
|
||
LANDING_TABLE_COLUMNS_DEVICE = [ | ||
('insert_id', 'NUMBER IDENTITY START 1 INCREMENT 1'), | ||
('snapshot_at', 'TIMESTAMP_LTZ(9)'), | ||
('raw', 'VARIANT'), | ||
('serial', 'VARCHAR(256)'), | ||
('address', 'VARCHAR(256)'), | ||
('name', 'VARCHAR(256)'), | ||
('network_id', 'VARCHAR(256)'), | ||
('model', 'VARCHAR(256)'), | ||
('mac', 'VARCHAR(256)'), | ||
('lan_ip', 'VARCHAR(256)'), | ||
('wan_1_ip', 'VARCHAR(256)'), | ||
('wan_2_ip', 'VARCHAR(256)'), | ||
('tags', 'VARCHAR(256)'), | ||
('lng', 'FLOAT'), | ||
('lat', 'FLOAT'), | ||
] | ||
|
||
|
||
def get_data(url: str, token: str, params: dict = {}) -> dict: | ||
headers: dict = { | ||
"Content-Type": "application/json", | ||
"Accept": "application/json", | ||
"X-Cisco-Meraki-API-Key": f"{token}", | ||
} | ||
try: | ||
log.debug(f"Preparing GET: url={url} with params={params}") | ||
req = requests.get(url, params=params, headers=headers) | ||
req.raise_for_status() | ||
except HTTPError as http_err: | ||
log.error(f"Error GET: url={url}") | ||
log.error(f"HTTP error occurred: {http_err}") | ||
raise | ||
log.debug(req.status_code) | ||
return req.json() | ||
|
||
|
||
def connect(connection_name, options): | ||
landing_table_client = f'data.meraki_devices_{connection_name}_client_connection' | ||
landing_table_device = f'data.meraki_devices_{connection_name}_device_connection' | ||
|
||
comment = yaml_dump(module='meraki_devices', **options) | ||
|
||
db.create_table(name=landing_table_client, | ||
cols=LANDING_TABLE_COLUMNS_CLIENT, | ||
comment=comment) | ||
db.execute(f'GRANT INSERT, SELECT ON {landing_table_client} TO ROLE {SA_ROLE}') | ||
|
||
db.create_table(name=landing_table_device, | ||
cols=LANDING_TABLE_COLUMNS_DEVICE, | ||
comment=comment) | ||
db.execute(f'GRANT INSERT, SELECT ON {landing_table_device} TO ROLE {SA_ROLE}') | ||
return { | ||
'newStage': 'finalized', | ||
'newMessage': "Meraki ingestion tables created!", | ||
} | ||
|
||
|
||
def ingest(table_name, options): | ||
ingest_type = 'client' if table_name.endswith('_CLIENT_CONNECTION') else 'device' | ||
landing_table = f'data.{table_name}' | ||
|
||
timestamp = datetime.utcnow() | ||
api_token = options['api_token'] | ||
whitelist = options['network_id_whitelist'] | ||
|
||
for network in whitelist: | ||
try: | ||
devices = get_data(f"https://api.meraki.com/api/v0/networks/{network}/devices", api_token) | ||
except requests.exceptions.HTTPError as e: | ||
log.error(f"{network} not accessible, ") | ||
log.error(e) | ||
continue | ||
|
||
if ingest_type == 'device': | ||
db.insert( | ||
landing_table, | ||
values=[( | ||
timestamp, | ||
device, | ||
device.get('serial'), | ||
device.get('address'), | ||
device.get('name'), | ||
device.get('networkId'), | ||
device.get('model'), | ||
device.get('mac'), | ||
device.get('lanIp'), | ||
device.get('wan1Ip'), | ||
device.get('wan2Ip'), | ||
device.get('tags'), | ||
device.get('lng'), | ||
device.get('lat'), | ||
) for device in devices], | ||
select=db.derive_insert_select(LANDING_TABLE_COLUMNS_DEVICE), | ||
columns=db.derive_insert_columns(LANDING_TABLE_COLUMNS_DEVICE) | ||
) | ||
log.info(f'Inserted {len(devices)} rows ({landing_table}).') | ||
yield len(devices) | ||
|
||
else: | ||
for device in devices: | ||
serial_number = device['serial'] | ||
|
||
try: | ||
clients = get_data(f"https://api.meraki.com/api/v0/devices/{serial_number}/clients", api_token) | ||
except requests.exceptions.HTTPError as e: | ||
log.error(f"{network} not accessible, ") | ||
log.error(e) | ||
continue | ||
|
||
db.insert( | ||
landing_table, | ||
values=[( | ||
timestamp, | ||
client, | ||
client.get('id'), | ||
client.get('mac'), | ||
client.get('description'), | ||
client.get('mdnsName'), | ||
client.get('dhcpHostname'), | ||
client.get('ip'), | ||
client.get('switchport'), | ||
client.get('vlan') or None, # vlan sometimes set to '' | ||
client.get('usage', {}).get('sent') or None, | ||
client.get('usage', {}).get('recv') or None, | ||
serial_number, | ||
) for client in clients], | ||
select=db.derive_insert_select(LANDING_TABLE_COLUMNS_CLIENT), | ||
columns=db.derive_insert_columns(LANDING_TABLE_COLUMNS_CLIENT) | ||
) | ||
log.info(f'Inserted {len(clients)} rows ({landing_table}).') | ||
yield len(clients) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry I wasn't more clear on this. could you please replace this db.insert expression with --
I think it will do the same thing but more succinctly.
to clarify
x or y
will returny
ifx
if "falsy", i.e. ifbool(x) is False
, anddict
's.get
returns None as the default value if the key is not presentThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, are you sure
vlan
needs to be treated differently thanip
orswitchport
? this seems off.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sometimes
client.get('vlan')
will return''
instead ofNone
, which cannot be inserted into the database since theVLAN
column requires a number. I like your suggested change though! I just can't treatvlan
the same way asip
because that '' value is invalid in our case.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, interesting. If you're sure that the others never return
''
then lgtm. Adding a comment like --might help to future folks, as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've never seen that occur with usage:recv or usage:sent, but I think it'll be best if we used the same approach for those instances too. Thank you for this feedback!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sgtm