MapQuest Data Manager is a powerful web application you can use to upload and maintain your own custom location data in a MapQuest-hosted database. By providing you with a facility to store your data on MapQuest servers, Data Manager frees you from having to maintain your own server to support whatever location-aware applications you are developing.
In this series of articles, I will introduce you to the basics of Data Manager by taking you through the development of a simple application called PhotoTrek.
What Is PhotoTrek?
As an avid photographer, I am always looking for ways to improve: improve the photos I take, improve the processing that inevitably follows, and--most recently--better calculate optimal routes from one "photo opportunity" to another. OK, so I hadn't really thought much about optimizing my route between locations until I started work on this article. But hey, it does provide a "real-world" scenario you can use to learn about Data Manager! Combine this with the fact that where I live (just outside of Washington D.C.), there is no shortage of really great locations to take pictures, and you have a cool new app--PhotoTrek--in the making!
In this article, you will use Data Manager to upload some custom locations to your own MapQuest-hosted database. In future articles, I will cover how to modify this data after it has been uploaded and show how to retrieve the data and use it as the basis for proximity searches and routing.
What You Need to Know
So, what do you need to know to upload data using Data Manager? Nothing more than how to create a text file and get around a web site--it's that simple! So let's get started by taking an initial look at Data Manager.
To get started, open your favorite web browser and go to http://developer.mapquest.com. To use Data Manager, you first need to register with the MapQuest Developer Network. After registering, you can find Data Manager on the Developer Network web site, after logging in. (Look in the upper-right corner of the web site for the Log In icon.) Then, click the Data Manager button on the blue menu bar stretched across the top of your browser window. This will take you to the Data Manager home page, shown in Figure 1.
Figure 1. Data Manager home page
Click Getting Started and take a minute or two to read through this section. It provides a great overview, and a link to the MapQuest Data Manager User Guide, which you'll likely find to be a valuable source of information as you proceed.
Getting Started with Data Manager
The first thing you should do before working with Data Manager is to ensure that you have set your working preferences. To do this, click the Preferences link on the left side of the Data Manager page to display the Preferences page:
Figure 2. Data Manager Preferences page
There are three fields you need to be concerned with on the Preferences page: Default Push Location Data to Production Setting, Push Ambiguities, and Email Notification Setting.
Default Push Location Data to Production Setting
This setting determines whether your location data is automatically "pushed"--that is, moved--from the staging database to the production database. You can choose not to push the data, to push the data if there are no errors, or to push the data regardless of whether there are any errors. Click the first (default) option, which is to leave the data in the staging database after uploading it. After you get comfortable using Data Manager (especially creating upload files) you might want to change this setting to have Data Manager push your location data to production, as long as there are no errors (the second option). Pushing uploaded data to production regardless of errors is typically not a wise thing to do.
This setting determines what Data Manager does when it is unable to resolve a location in your upload file to a unique latitude and longitude (Lat/Long); that is, when more than one match was found for the data you specified. Because ambiguities typically indicate insufficient information in your location file, it is best to choose the first option, which is to not push these records to the production database.
E-mail Notification Setting
Uploading your file can take a bit of time depending on how many records you have included, in addition to other factors. Because of this, Data Manager allows you to specify an e-mail address where you will receive notification when your upload is complete.
Default Table Selection
You do not need to make any changes to this field. In fact, you can't change it if this is the first time you are using Data Manager, because there are no tables to choose from until you create them! You can always return to the Preferences page later and make a selection here to specify which table is made "current" each time you start up a new session.
After verifying your settings, click Save to save your preferences.
Creating a Custom Table
Now that you have established your working preferences, you need to create a table on the MapQuest server that will hold your custom location data. Table creation is an interactive process in which you add, delete, and rearrange fields. To access it, click the Tables link on the Data Manager home page. Note that the first time you click Tables, Data Manager takes you directly to the Add Table page, shown in the following image.
Figure 3. Adding a new table
Before creating a table to hold the PhotoTrek location data, you need to identify the information you want to store. In addition to the standard fields that are automatically part of any table you create using Data Manager, in this case, you'll add two custom fields:
- AssociatedURL String
- LocationType Numeric
AssociatedURL contains the URL (if applicable) of the location of your "photo opportunity." LocationType identifies the type of location specified--for example, historic landmark, museum, or statue. You'll use this field in future articles; for now you will simply populate it according to the following data table:
01 - Museum
02 - Statue
03 - Historic Landmark
04 - Sports Arena/Stadium
05 - Church/Cathedral/Temple/etc.
You can (and probably will) add more location types in the future, but this will suffice for now.
Although table creation can be done all at once, you'll break it down into a two-step process here. First, create the table with the standard Data Manager-provided fields. Second, you'll add the two custom fields identified above.
To create the PhotoTrek table, type
PhotoTrekPOIs in the Table Name Suffix box, and then click Submit.
Figure 4. Creating a table
If you click Submit at this point, you instruct Data Manager to create a new table for you, using the table name suffix as part of the table name. In practice, you can think of the table name suffix as the table name itself; use whatever name is meaningful to you. When Data Manager creates the table, it appends the suffix you specified to the constant "MQ_" and the client ID you were assigned when you registered on the site. For instance, if you were assigned client ID "9999," then the newly created table would be called "MQ_9999_PhotoTrekPOIs."
After creating the table, Data Manager takes you back to the Tables page, only this time your newly created table shows up in a list, along with a number of icons you can use to print a schema, or to edit, delete, or clone your table.
Note: I have hidden my client ID for the purpose of this article. Your assigned client ID will show up as part of the table name.
Figure 5. Table created and ready to edit
Now that you have created your custom "PhotoTrekPOIs" table, you can go in and add the two additional fields that you will use to store the location's URL, and its type. To edit a table's schema, click Edit Table on the Tables page. (Note that if you have more than one table, you can click the table you want to edit in the Change Active Table drop-down list.)
Before you click Edit Table, I want to emphasize the importance of keeping a local copy of data to upload on your own computer. Editing the schema of a table in Data Manager will erase all the data that is currently in the table, requiring you to upload the data again. Obviously, if you rely upon MapQuest to keep the only copy of your data, you will have no recourse if you edit the table's schema!
With that said, go ahead and click Edit Table, and then click OK when you see the warning about losing data currently in the table.
The first thing you will notice is that there are already a number of fields in your schema. These are fields that Data Manager automatically places in your table for you, some of which are required; I will cover these in more detail in the following section.
Then, in the Field Name box, click AssociatedURL; in the Field Type box, click String, and then click Add. Note that this does not actually add the field to the table--this doesn't happen until you click the Submit button, after adding all of your fields.
Figure 6. Adding the AssociatedURL field
Next, in the Field Name box, click LocationType; in the Field Type box, click Integer, and then click Add.
It is a good idea to make sure that the fields have been added, before you submit your changes. You can do this by dragging the scroll bar down on the right side of the field list. After verifying your new fields are there, click Submit.
Figure 7. Verifying added fields
Preparing Data for Upload
Now that you have created a custom table on the MapQuest server to hold your location data, your next step is to prepare a batch text file that contains the data you want to upload. The batch file is a comma-delimited file that contains each field in quotes. I used a simple text editor to create mine. Whatever you choose to use, make sure it creates a simple text file with no embedded control characters.
Before you create your file, take a moment and look at the predefined fields Data Manager automatically placed in your table because these affect the layout of your batch file. Recall that the data you are about to upload is location data; that is, it specifies a physical location here on planet Earth, ultimately represented by a unique latitude and longitude. Lucky for you, you don't need to know in advance the Lat/Long of any of the locations you plan to upload. Why? Well, because Data Manager will "geocode" your locations! Geocoding is the process of assigning a unique Lat/Long to a location, and is explained in chapter 2 of the Data Manager User Guide, if you want to know all the details.
What this means here is that you only need to provide sufficient location information--for example, a street address--in your batch file, and Data Manager will do the rest. That said, following is a table of the required fields that must appear in your import file. A detailed explanation of each is in chapter 5 of the Data Manager User Guide (hint, hint!).
Notice that the fields in the table generally correspond to the standard fields that are part of the table layout provided by Data Manager (shown in Figure 3) when you add a new table.
Table 1. Import File Format
||This contains a unique ID for each location record
||A description of the location
||Street number and name
||The location's city
||The location's county
||2-character state/province code (blank if outside the U.S. or Canada)
||The ZIP Code/postal code of the location
||2-character ISO Country Code
||- optional -
||- optional -
||- optional -
||- optional -
||- optional -
|User Fields (1-10)
||- unused -
|Search Fields (1-20)
||- unused -
|Text Fields (1-100)
||- unused -
|Addt'l Fields (1-100)
In this exercise you will create an extremely basic file of only two location records. (I will cover the "how to" of adding more records in a future article.) The two locations I'm using are excellent spots that I have personally photographed, which is why they are going in the file; feel free to substitute your own!
Given the table of fields, in conjunction with the custom fields (AssociatedURL and LocationType) you added to your table, your import file ends up looking like this:
"0000000010","United States Botanic Garden - National Garden (Regional Garden)",
"Maryland Ave SW & 3rd St SW","Washington","District of Columbia","DC","20224",
"0000000020","Civil War Statue","100 Maryland Ave SW","Washington",
"District of Columbia","DC","20224","US","","","","","","","2"
Notice a few important things about this import file:
- I have arbitrarily assigned Record IDs (the first field) for these two records of "0000000010" and "0000000020." These can be anything you want, as long as they are unique.
- I have specified a complete street address for these two locations to increase the odds that geocoding will be successful.
- I have provided empty quotes for optional "standard" fields (fields 9-13).
- The "unused" fields (14-16) are not included in the import file at all.
- You are using the first two out of a possible 100 "Additional Fields" to provide the two custom fields "AssociatedURL" and "LocationType" (remember that you added these earlier when you created your table?).
Now that you have created your import file, the next step is to instruct Data Manager to upload it. Click Upload File on the Data Manager home page to go to the Upload File page.
Figure 8. Uploading a file
The Upload File page is fairly straightforward. Type the name of the upload file you created (or browse to it by clicking the Browse button), leave the Delimiter box set at Auto, and leave Push to Production set to the first option. Click the Upload button, and Data Manager will upload your file to a temporary staging area, and will send you an e-mail when it is finished.
Note that you will not be able to do any maintenance on this particular table while Data Manager is processing your upload in the background.
Verifying Your Upload
After you receive an e-mail indicating that processing is complete, you can view a summary of the uploaded file on the Data Summary page, which you can access from the Data Manager home page.
Figure 9. Data upload summary
The Data Summary page is an excellent way to quickly determine the disposition of each location processed in your uploaded file. Because you are considering data in the "staging" area, don't click the Production link at the top of the page at this point; this would display data in the production area, in which you don't yet have any data.
The first block of statistics displays the "quality" of the geocode match that was made for each location. In this case, one record was geocoded at the level of a street address, and the other at the level of an intersection, which exactly matches the location data specified (one was a street address, the other was an intersection.) The second block of statistics displays the number of ambiguous matches--ones that have found multiple geocodes--for each location processed; there are none in this instance. Not Geocoded identifies the number of records that were tagged with a valid Lat/Long in the input file, and therefore Data Manager did not geocode. Invalid (Processed) identifies the number of matches that contained invalid records yet were still processed, and Not Processed (Errors) identifies records that could not be processed due to invalid location information.
One nifty feature you can access from this page are other pages on which you can view and edit data that has been uploaded. Simply click one of the links in the Geocode Quality column, and you will be presented with a list of all the uploaded records that fit that particular disposition category--for example, Address Quality. When the list is displayed, you can change the sort order of the records, edit them, move them around, or even delete them.
Figure 10. Editing uploaded data
Because all of your records (all two of them!) processed successfully, go ahead and push them to production so that you can start using them.
Pushing Data to Production
You can push data to production directly from a link on the Data Summary page, or from the Production link on the Data Manager home page. Data Manager will ask you to verify that you are ready to push your data to production, and will indicate that any data currently in production will be overwritten with the data currently in staging.
Figure 11. Push data to production
Click Yes to proceed with the data push. When it is complete, you will receive a notification e-mail. As with your initial upload of data, the current table is inaccessible until the push is complete.
After you receive the e-mail indicating your push to production is complete, it is a good idea to view the statistics, to verify the data push. You can view this information on the Data Summary page; click the Production link at the top to view the results.
In my next article, continuing the development of PhotoTrek, I'll show you how to use Data Manager to make changes to already-uploaded data using incremental batch files, in addition to adding and editing locations directly using forms on the Data Manager web site. I will also cover some of the basic table maintenance options in Data Manager, and review the options Data Manager provides for downloading data.
See you then!