Create APIs for Aggregations and Joins on MongoDB in Below 15 Minutes

Rockset has teamed up with MongoDB so you’ll be able to construct real-time apps with knowledge throughout MongoDB and different sources. In case you haven’t heard of Rockset or know what Rockset does, you’ll by the tip of this information! We’ll create an API to find out air high quality utilizing ClimaCell knowledge on the climate and air pollution.

Air high quality has been documented to impact human well being (sources on the backside). Specifically, ranges of particulate matter (PM), sulfur dioxide (SO2), nitrogen dioxide (NO2), the ozone molecule (O3), and carbon monoxide (CO) are measured with a view to counsel an individual’s really useful exercise degree outdoor. What results PM, SO2, NO2, O3, and CO ranges are topic to in depth research: scientists look at temperature, humidity, site visitors congestion, wind gust, and so forth to see how the air high quality index (AQI) modifications with respect to those pollution.

It’s vital to notice that it is a pattern app to indicate how MongoDB can combine with Rockset and demo Rockset’s tremendous powers of constructing APIs. This isn’t a scientific challenge that’s meant to be exhaustive in its conclusion. Rather more knowledge is required, and lots of variables will not be considered when doing the question. For extra on air air pollution, see under on sources. To leverage Rockset and MongoDB on your real-time purposes (scientific or not), proceed studying!

In subsequent tutorials, I can present how this dataset can probably be used to know how temperature and humidity impacts AQI of PM and O3. You’re additionally welcome to do the challenges on the finish of the tutorial to earn Rockset credit score, swag, and extra!

On this information, we’re going to:

1. Combine MongoDB Atlas and Rockset

   • Construct collections on MongoDB that can every map to Rockset’s collections
2. Construct a Python script that can constantly get the climate and air high quality knowledge from ClimaCell (Powered by ClimaCell) and put it into MongoDB
3. Use Rockset’s Question Editor to write down a question on real-time knowledge coming in from MongoDB
4. Create an API to the question we wrote
5. Execute the API in our app and print out a press release in regards to the air high quality

Setup

Let’s get began on MongoDB

1. After you’ve created an account on MongoDB Atlas, go forward and navigate to Database Entry → Customized Roles and click on on Add New Customized Function. The picture under is what actions and roles ought to be added for rockset-role. Rockset has safe read-only entry to MongoDB Atlas.

2. Navigate to the Database Customers tab and click on on Add New Database Consumer. Keep in mind the password right here, as a result of we might want to use it once more, after we create an integration on Rockset.

3. Go forward and create one other database person that has Atlas admin privileges. We shall be utilizing this database person in our Python app. You possibly can title this person yourName-admin. Ensure you bear in mind the password right here, as a result of we are going to use this in our Python app.
4. Navigate to the Community Entry click on on Add IP Tackle and whitelist these IPs:
   Embedded content material: https://gist.github.com/nfarah86/c6014ea1d60ec6113948d889afb16fdf

5. Navigate to Clusters and click on on Collections then click on on Create database. We’re going to create a weather_pollution_db and it’s going to have weather_data assortment.
   
   
   
6. Below the weather_pollution_db, there’s going to be a plus signal. Click on on the plus signal, and add one other assortment, air_pollution_data.

7. Return to Clusters and click on on Join and click on on Join your software. Copy the string, as a result of we are going to use it in Rockset. After we are in our Rockset account, the username is rockset-user and the password is the password you used once you created rockset-user. In our Python app, the username shall be yourName-admin and the password related to that database person.

8. That’s it for MongoDB! Let’s go forward and write our Python app!

Let’s construct our Python app

1. Create a challenge folder, and in that challenge, create a file .env.

2. Within the .env file add this:

   • Mongo_URI=“<uri string>”
   • The "<uri string>" is your connection string from MongoDB. Ensure you change the username and password within the connection string with yourName-admin and the password you used once you created that database person.
   • It ought to look one thing like this:
     mongodb://nadine-role-admin:password....

3. In case you use a virtualenv go forward activate a env for this challenge. Ensure you’re below Python 3.7 or greater.

   • I personally use Pyenv, however be at liberty to make use of no matter you need!

4. Set up python-dotenv:

   • $ pip set up python-dotenv

5. Set up [pymongo] and [dnspython]:

   • $ pip set up pymongo
   • $ pip set up dnspython==1.16.0
6. Inside our challenge folder, go forward and create settings.py This file ought to appear to be this: Embedded content material: https://gist.github.com/nfarah86/f87a9d37f1f72bb2d4a73d9b73dc87b4.
7. Create one other file within the challenge folder known as mongo_config.py. It ought to appear to be this: Embedded content material: https://gist.github.com/nfarah86/1fc7bc9987d27edbec0fa9b32be95163
8. Within the challenge folder, go forward and create file known as script.py. All we’re going to do is make certain our Python app is connecting to MongoDB: Embedded content material: https://gist.github.com/nfarah86/4d8e87ff6e70e1da1c017e80b8daeef2
9. Below Clusters, click on on the collections button. Go to weather_pollution_db and click on on weather_data. It’s best to see this:

10. Now that we all know we are able to insert knowledge into MongoDB, let’s go forward and create a ClimaCell developer account and get an API KEY.

11. In settings.py go forward and add this:

    • CLIMACELL_API_KEY = os.environ.get('CLIMACELL_API_KEY')
12. I selected ClimaCell as a result of they provide realtime knowledge for climate and air air pollution. We’re going to work with this api. They’ve completely different parameters that may be added to the request. You possibly can discover these right here.

13. In our challenge folder go forward and pip set up a number of libraries:

    • $ pip set up requests
    • $ pip set up timeloop

14. In script.py go forward modify the packages we’re going to make use of: Embedded content material: https://gist.github.com/nfarah86/a49cbaa033239c636ef4f3bbe1dca2d0

    • Timeloop a library that may run jobs at designated intervals.

15. Hold insert_to_mongo() and add this operate in script.py to get the climate knowledge: Embedded content material: https://gist.github.com/nfarah86/d2e3cc9236547e2fa630fd368dfee994

    • That lat and lon correspond to Beijing.
16. Now, we’re going so as to add this operate to get the air high quality: Embedded content material: https://gist.github.com/nfarah86/c598dbea0274d43215f15c9f01eca672
17. We’ll modify insert_to_mongo() to appear to be this: Embedded content material: https://gist.github.com/nfarah86/e43f4ad2d8f7e3ca4b8d761408bc853c
18. To ensure we’re operating constantly, write this: Embedded content material: https://gist.github.com/nfarah86/959d875ad5ffcc08e16e3bf25358385a
19. After, write foremost() like this: Embedded content material: https://gist.github.com/nfarah86/831e295b663aceb93603d9986c815b43
20. This is a gist of what your script.py ought to appear to be: Embedded content material: https://gist.github.com/nfarah86/85caee5b14639e238e34715094cc5436

21. Now, run:

    • $ python script.py to populate MongoDB.
22. Whereas the script is operating, let’s get began on Rockset.

Let’s get began on Rockset

1. Login to Rockset and navigate to the Integrations tab on the left. Click on on Add Integration. Click on on MongDB and click on on begin:

2. Examine the primary field MongoDB Atlas. We’re going to call this integration Rockset-Mongo-Integration. For the username and password, go forward and put rockset-user and the password you utilize once you created this database person. Paste the connection string within the subsequent field and click on on Save Integration.

   • Every integration can be utilized to entry a number of databases and collections in the identical MongoDB cluster

3. Below Collections click on on Create Assortment. Choose MongoDB because the supply.

4. Click on on the rockset-mongo-integration.

5. We’re going to call our new assortment on Rockset weather_data_collection. This isn’t tied to MongoDB. Go forward and fill out the remainder of the web page with the database and assortment we created on MongoDB. We’re going so as to add 2 collections, however let’s begin with the weather_data from MongoDB.

   • You see, Rockset is in a position to hook up with MongoDB. You possibly can confirm what knowledge shall be ingested into the Rockset assortment on the right-hand aspect. If you’ve created a group and operating a data-driven app in real-time, Rockset will constantly sync with MongoDB so your knowledge can have the most recent data.
   • Let’s click on Create on the backside.

6. Observe the identical steps, step 3-5, to create the gathering, air_pollution_data_collection. On the finish, it ought to appear to be this:

• Observe permissions will be modified within the MongoDB UI at any time with out the necessity to drop and recreate integration. Besides when username and/or password or connection string modifications – then the person might want to drop and recreate the Rockst integration

Assemble a Question on Rockset

1. On the left bar, let’s navigate to the Question Editor.

2. On the tab if we write:

   • Choose * from commons.air_pollution_data_collection; we must always see some output:

  - Go forward and do that for the `weather_data_collection`

3. We’re going to write down this pattern question: Embedded content material: https://gist.github.com/nfarah86/2d9c5bc316d55cfd0fcf17b4ded9141f

   • We’re averaging the PM10 knowledge and the climate temperature knowledge. We’re going to hitch each of those collections based mostly on the date. In case you seen the timestamp within the JSON, the date is in ISO 8601 format. In an effort to be part of on the times (and eliminates the minutes, hours, and seconds), we’re going to do an extraction.
4. Run the question.

5. After we run this question, we wish to embed it in our app, so we are able to notify our customers when the degrees fluctuate, and probably predict, based mostly on climate, what PM10 ranges could appear to be the subsequent day.

   • We’re going to wish much more knowledge than what we’ve now to foretell based mostly on temperature, however it is a begin!

Construct an API on our question on Rockset

1. On the highest nook, click on on Create Question Lambda. A Question Lambda is a approach to make an API endpoint to the SQL question you write. Within the Python app, we gained’t have to write down client-side SQL, stopping safety vulnerabilities.

2. Give your Question Lambda a reputation and outline. Afterwards, it is best to see some code snippets on the subsequent display.
3. Let’s navigate again on the Question Editor and write one other question to get present climate in a brand new tab. Generally we could get a null subject, so let’s go forward and write this within the Question Editor: Embedded content material: https://gist.github.com/nfarah86/4581c6bc09d30045ae75a5f330a8d72f
4. Create one other new Question Lambda.
5. If we wish to seize the code snippet or URL of the Question Lambdas we simply created, navigate on the left aspect menu to Question Lambda and click on on the lambda you created.

Execute APIs on our app

1. When you create a Question Lambda, you’ll see one thing like this:
   
   
   
   

2. There are two methods I’ll present how we are able to execute a lambda:

   • Make an HTTP Request
   • Rockset’s Python shopper (backside field the place my API is boxed out)

3. Make an HTTP Request:

   • Let’s go forward and make an HTTP request to get the current_weather knowledge. Listed here are the steps to do that:
   • Go forward and set your ROCKSET_API_KEY in your .env. Import it in settings.py like we did earlier than.
   • On Rockset, navigate to the Question Lambda that has the current_weather question. Copy the question lambda endpoint.
   • We’re going to write down this operate that can make an HTTP request to that endpoint:Embedded content material: https://gist.github.com/nfarah86/3a0ef9b1524532247e3ea7c504489d23

4. Use the Rockset Shopper to ship a request:

5. Then, we’re going to show the outcome:Embedded content material: https://gist.github.com/nfarah86/a0d1e15319bc117ef55ce35187fb6480
6. We’re going to alter sample_job_every_120s()so as to add make_requests so we are able to execute the Question Lambdas and show the information:Embedded content material: https://gist.github.com/nfarah86/0a54e082c9026aa5c9940b24836d9c65
7. Write make_requests() so it seems like this:Embedded content material: https://gist.github.com/nfarah86/dea06329b25887bb58a0ef74c4a12fb0
8. After you run the script, it is best to see this:Embedded content material: https://gist.github.com/nfarah86/32b35bd3269fbd1701dc57252fa783e4
9. That’s it! This wraps it up for the MongoDB-Rockset Python App!

Mission Code

You could find the total challenge, together with the SQL statements right here. In case you have questions in regards to the challenge, Rockset, or MongoDB, you’ll be able to attain out in our group.

Assets:

Different MongoDB sources: