Add ASN maxmind database integration #604

XAMPPRocky · 2022-09-26T15:07:31Z

This adds a Maxmind DB integration so that if you provide quilkin with the source of a maxmind database that matches the structure of IPNetDB, it will output info about the ASN of a new session source.

I also refactored the admin server to be in cli to reduce duplication between run and manage.

closes #450

markmandel · 2022-09-26T22:55:34Z

Looks like it's unhappy about the MPL, if you add in the exception here:

quilkin/deny.toml

Lines 30 to 33 in 6a42df8

    
           exceptions = [ 
        
               # Each entry is the crate and version constraint, and its specific allow 
        
               # list 
        
           ]

Please also set dependecies in

quilkin/image/archive_dependencies.sh

Line 29 in 6a42df8

dependencies=()

to: (webpki-roots) so we capture the source in the image.

markmandel · 2022-09-27T23:35:44Z

src/cli.rs

@@ -91,12 +103,16 @@ impl Cli {
        );

        let config = Arc::new(Self::read_config(self.config)?);
-        if self.no_admin {
+        let _admin_task = if let Some(mode) = self.command.admin_mode().filter(|_| self.admin) {


Nice centralisation 👍🏻

markmandel · 2022-09-27T23:37:14Z

src/config.rs

+            management_servers: <_>::default(),
+            proxy: <_>::default(),
+            version: <_>::default(),
+            maxmind_db: Slot::empty(),


Now I'm curious, is there a publicly accessible maxmind db somewhere we could reference by default? 🤔 or licencing / security nightmare?

There might be, but I haven't found one. The only one I know of is IPNetDB, which is $300/a year.

Hi, I stumbled over this discussion with an alert. I'm a founder of the company that releases IPNetDB. If helpful, as this is an open source project you're welcome to build against, test and develop with IPNetDB databases without paying anything and you can redistribute it with an open source project just with attribution. The paid licence is only required if you launched a for-profit service with an IPNetDB database embedded in it. You're welcome to ask if you want clarification on any usage. The licence is not particularly restrictive.

We also have a mock mmdb database we use for testing internally which I can send over if you would prefer something to test with that has absolutely no licence but the same encoding format.

@meeb Thank you for your response! To clarify, would a paid licence be required if this open source project embedded with IPNetDB communicated with a for-profit service over the network? This project is a UDP proxy primarily for games, which are often commercial projects. So long as that use case can make use of IPNetDB without the paid licence, I wouldn’t have a problem integrating with it directly (though it also needs @markmandel approval). I just wouldn’t want the primary use-case of the project to have surprising licencing for our users.

As it currently stands with the current bare-bones licence options, maybe if the end result was a paid-for SaaS or PaaS. As Quilkin sits between client applications and game servers (a similar comparable service architecture would be HAProxy but for UDP game traffic?) would I be correct in saying what IPNetDB would do in for this project is add additional metadata to connections as they pass through the proxy?

Having said that, we release IPNetDB for people to use so I'm happy to have a discussion about your specific use case. We have organised redistribution licences, as well as granted specific use licences to projects in the past.

There are other considerations as well such as the database updates weekly which given the fluid nature of the internet some updates will likely be required. If auto-updating your proxy would download from our CDN which creates logs with client IPs in etc. so there might be some further legalese to approve if you do embed IPNetDB with an auto-updating feature.

Given this is likely going somewhat off topic for the purpose of this PR you're welcome to email me at licence@ipnetdb.com with a mention you're XAMPPRocky (or markmandel) from GitHub and I'll pick it up. I can also send you a small test mmdb file you can embed for your test suite as well if you like.

@meeb thanks so much for the offer!

A good first step for us would be to get our Open Source Programs Office to review the licences that are publicly available and get their approval. @meeb - is this the correct link, or is there something more comprehensive we should be looking at?

@markmandel yes those are the minimal licences. They're specifically brief to make it easy to use the database, however I would suspect you might want a slightly more robust licence if you wanted to redistribute the embedded database. Happy to chat to your OSPO. Given we allow free download of the databases the licence was always more trust based so we didn't bother with anything extensive to agree to. Assuming the use remains in an open source project I would suspect a simple attribution (and probably letting your users know their client may download data from our CDN periodically if the feature is enabled) would be sufficient. Feel free to pass on the above email address to anyone relevant internally to discuss it.

Sweet - thanks - will pass this along and swing back around when I get a resolution.

Just letting you know - I have my 1:1 tomorrow with my OSPO contact, I'll bring it up then, and let you know how it goes.

markmandel · 2022-09-27T23:37:19Z

src/proxy/sessions.rs

            dest: args.dest,
            created_at: Instant::now(),
            expiration,
            shutdown_tx,
        };
        tracing::debug!(source = %s.source, dest = ?s.dest, "Session created");

+        if let Some(mmdb) = &*crate::MaxmindDb::instance() {


Looks good - just so I'm super clear, we're only tracing this information on Session creation? (Which means this can't be a filter either, which is fair enough. i did wonder at one point if #586 should maybe have a "Session creation Filter/Plugin" - which this could have leaned into, but not sure if that's a good idea).

Do you think we'll want to also attach this information to any other traces that may come through this particular session?

Only other thought: We need some docs please 😃

Yeah for now I don't see much point in having it as a filter, as it's something that you want to only happen to successful connections, which is easy to guarentee here. We can always change it later.

Do you think we'll want to also attach this information to any other traces that may come through this particular session?

I think we can if we find it useful. Right now I'm just starting with Quilkin outputting this info and then trying to use it, and seeing what it needs from there.

Only other thought: We need some docs please 😃

Added

markmandel · 2022-09-27T23:37:36Z

src/proxy/sessions/manager.rs

@@ -98,6 +97,7 @@ impl SessionManager {
            .count();

        if expired_keys != 0 {
+            tracing::debug!("pruning expired sessions");


Nice cleanup.

quilkin-bot · 2022-10-10T19:08:09Z

Build Succeeded 🥳

Build Id: f31f8c00-8e57-4e35-aa7c-7515ff9a9465

The following development images have been built, and will exist for the next 30 days:

us-docker.pkg.dev/quilkin/ci/quilkin:0.4.0-dev-bb74296

To build this version:

git fetch git@github.com:googleforgames/quilkin.git pull/604/head:pr_604 && git checkout pr_604
cargo build

XAMPPRocky requested review from markmandel, luna-duclos, rezvaneh, vallfors and bombsimon as code owners September 26, 2022 15:07

github-actions bot added the size/m label Sep 26, 2022

markmandel reviewed Sep 27, 2022

View reviewed changes

markmandel approved these changes Sep 28, 2022

View reviewed changes

XAMPPRocky force-pushed the ep/asn-tracing branch 8 times, most recently from eceda6d to 60005c3 Compare October 10, 2022 18:32

Add ASN maxmind database integration

bb74296

XAMPPRocky force-pushed the ep/asn-tracing branch from 60005c3 to bb74296 Compare October 10, 2022 18:51

XAMPPRocky merged commit 02f8156 into main Oct 10, 2022

markmandel deleted the ep/asn-tracing branch October 21, 2022 20:58

markmandel added kind/feature New feature or request area/operations Installation, updating, metrics etc labels Nov 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ASN maxmind database integration #604

Add ASN maxmind database integration #604

XAMPPRocky commented Sep 26, 2022

markmandel commented Sep 26, 2022

markmandel Sep 27, 2022

markmandel Sep 27, 2022

XAMPPRocky Sep 28, 2022 •

edited

Loading

meeb Oct 1, 2022

XAMPPRocky Oct 1, 2022

meeb Oct 1, 2022

markmandel Oct 3, 2022

meeb Oct 4, 2022

markmandel Oct 4, 2022

markmandel Oct 6, 2022

markmandel Sep 27, 2022

XAMPPRocky Oct 10, 2022

markmandel Sep 27, 2022

quilkin-bot commented Oct 10, 2022

Add ASN maxmind database integration #604

Add ASN maxmind database integration #604

Conversation

XAMPPRocky commented Sep 26, 2022

markmandel commented Sep 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XAMPPRocky Sep 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quilkin-bot commented Oct 10, 2022

XAMPPRocky Sep 28, 2022 •

edited

Loading