Note
This technote is not yet published.
1 The Scope of the Document¶
This technical note documents a parallel procedure and tools developed for loading the SDSS/Stripe82 catalogs into PDAC Qserv. The catalogs were produced in a course of the LSST Summer 2013 Data Challenge effort which is covered in details at:
- https://dev.lsstcorp.org/trac/wiki/Summer2013
- https://confluence.lsstcorp.org/display/DM/Properties+of+the+2013+SDSS+Stripe+82+reprocessing
Note that the current document won’t provide any further explanation on how the catalogs were put together and prepared for the loading beyond the bare minimum info which is related to a location and a general structure of the input files used by the loading tools. The catalog preparation topics are fully covered in the following JIRA tickets:
- https://jira.lsstcorp.org/browse/DM-6905 - Locate the test dataset for PDAC
- https://jira.lsstcorp.org/browse/DM-7007 - Investigate coverage of S13 databases found so far
- https://jira.lsstcorp.org/browse/DM-7053 - Assemble a complete database with S13 DRP catalog
- https://jira.lsstcorp.org/browse/DM-7498 - Begin ingest of S13 DRP data to PDAC
2 Introduction¶
Here be a brief overview of the loading procedure, tools (GitHub), etc.
3 Explain the environment¶
- read-only GPFS
- explain how to start/stop/inspect Qserv
- explain where to find log files
4 Input datasets and configuration files¶
- located on GPFS
- explain what’s found in there and how it was produced (links to the JIRA tickets explaining the merge procedure)
5 Tools needed¶
- sudo yum installing required dependencies on the nodes
- install stack
- install a specific ticket & build it
- replicate stack across all nodes
6 Configuring Qserv¶
- Creating database on all nodes and granting privileges
- setting up WMGR secret file on all nodes
- configuring CSS on Master
- configuring MySQL in worker nodes - adding the database name to this table:
SELECT * FROM qservw_worker.Dbs;
Table 1 Result set¶ db sdss_stripe82_00
7 Loading tables¶
7.1 Non-partitioned tables¶
- a special case of _NoFile table
7.2 Deep Forced Sources¶
- the director table of the Qserv catalog
- explain steps
- explain the performance of the process
7.3 Deep Sources¶
- explain steps
- explain the performance of the process
8 Testing¶
- log to the Master node
- how to connect to the mysql-proxy process
- queries
9 Cleaning up and starting from scratch¶
- explain all steps needed to reset Qserv