Replication monitor

Once per hour a segment database at each site is queried to collect a list of segments corresponding to LFNs published during the last two days. At the observatories segments corresponding to raw frames from /frames and /fb0_frames are also computed by directly scanning the disks (to check if one can trust segment database for a given time interval or not everything has been published yet: disk2disk script, LDAS, publishing scripts, replication could be down). The exact end time of the considered time interval is chosen as follows: (current time - 300 seconds)/60*60. The integer division is used to minimize the difference in current times of script execution at different sites (the scripts run as cron job at each site and depending on the load on gateways 'current time' might differ by a second or two between the sites). 300 second offset is used to allow for possible delays in replication between sites. Below this time segment is called query segment. To be able to compare the list of replicated and original segments, we would like in ideal to have the same query segment at all the sites. Once per hour at each site a script runs that collects the output from the above query from all 3 sites, compares the published segments between the sites and generates at each site a web page (which should be the same unless there are network problems): LHO, LLO, CIT. One might also find it useful to get the latest pickled objects (LHO.out, LLO.out, CIT.out) of a class containing all the information published on this web page. To be able to use those pickled objects in your python program, import a stripped down version of the Stats class before loading the objects. The are also summary pickled Stats objects that provide the same information for the whole run up to the time the script ran last time (summary_LHO.out, summary_LLO.out). The summary objects are obtained not by querying the database (that takes most of the day for the whole S5) but by adding a new SITE.out object (covering last two days) to the previous summary object summary_SITE.out (that covers last two days minus an hour). Since the segments in Stats objects are already coalesced, this is an instantaneous operation.

The information is organized in a table with a row corresponding to a replication source and a column corresponding to a replication target. For example, LLO row compares L-R-* LFNs published at LLO and other sites.

Diagonal elements in the table contain the following information:

Query segment: GPS start, end times, duration.
Same query segment in a more readable format.
Coalesced segment list of LFNs that got inside the query segment (LFN segments):
select start_time, end_time from LFN where start_time > query_startTime and end_time < query_endTime and name like 'site-R-%'
Each segment has index, start, end, duration.
Total duration of the above LFN segments.
Gap segments: difference between query segment and LFN segments. It is OK to have gaps of duration less than 32 seconds at the beginning and end of the query segment. Other gaps indicate either gaps in real data or gaps in what was published; such gaps are shown in red to indicate possible problems.
Total duration of the above gaps.
Intersection of a union of segments covered by raw frames from /frames and /fb0_frames with a query segment.

Offdiagonal elements contain the following information:

Difference between query segments of replication source and target sites; it should be 0,0 in ideal, if it is not, the corresponding line is shown in yellow.
Difference in total duration of LFN segments between the replication source and target sites; if the difference is not zero, the corresponding entry is shown in red.
Difference between the replication source and target LFN segments; if there is no difference, the line is shown in green, if there is a difference, it is shown in red. This is the most important entry in the table that indicates if replication is working or not.

-->LHO

-->LLO

-->CIT

LHO-->

877875900,878048700,172800;

Oct 31 2007 07:24:46 PDT, Nov 02 2007 07:24:46 PDT

1,877875904,878048512,172608;

172608

1,877875900,877875904,4;

2,878048512,878048700,188;

192

1,877875900,878048700,172800;

0,0

LLO-->

0,0

877875900,878048700,172800;

Oct 31 2007 09:24:46 CDT, Nov 02 2007 09:24:46 CDT

1,877875904,878048416,172512;

172512

1,877875900,877875904,4;

2,878048416,878048700,284;

288

1,877875900,878048700,172800;

0,0

CIT-->

0,0

877875900,878048700,172800;

Oct 31 2007 07:24:46 PDT, Nov 02 2007 07:24:46 PDT

1,877875900,878048700,172800;

172800

Unpublished raw frames for the whole S5 run

The table below shows the difference between segments corresponding to raw frames from /frames, /fb0_frames and /archive from the site of the origin and the segments corresponding to the published (in the segment database of the raw frame origin) LFNs.

Site	GPS start	GPS end	duration	Date start	Date end
H	878045088	878049120	4032	Nov 02 2007 08:24:34 CDT	Nov 02 2007 09:31:46 CDT
L	874616576	874617152	576	Sep 23 2007 16:02:42 CDT	Sep 23 2007 16:12:18 CDT
	875969856	875985920	16064	Oct 09 2007 07:57:22 CDT	Oct 09 2007 12:25:06 CDT
	876082560	876082592	32	Oct 10 2007 15:15:46 CDT	Oct 10 2007 15:16:18 CDT
	876183776	876187904	4128	Oct 11 2007 19:22:42 CDT	Oct 11 2007 20:31:30 CDT
V	864147600	864151200	3600	May 25 2007 11:59:46 CDT	May 25 2007 12:59:46 CDT
	867798000	867815984	17984	Jul 06 2007 17:59:46 CDT	Jul 06 2007 22:59:30 CDT
	867827040	867862800	35760	Jul 07 2007 02:03:46 CDT	Jul 07 2007 11:59:46 CDT
	870591600	870593888	2288	Aug 08 2007 01:59:46 CDT	Aug 08 2007 02:37:54 CDT
	870613200	870628736	15536	Aug 08 2007 07:59:46 CDT	Aug 08 2007 12:18:42 CDT
	870635808	870636016	208	Aug 08 2007 14:16:34 CDT	Aug 08 2007 14:20:02 CDT
	870640128	870657120	16992	Aug 08 2007 15:28:34 CDT	Aug 08 2007 20:11:46 CDT
	870661488	870661984	496	Aug 08 2007 21:24:34 CDT	Aug 08 2007 21:32:50 CDT
	870663760	870709184	45424	Aug 08 2007 22:02:26 CDT	Aug 09 2007 10:39:30 CDT
	870715856	870764848	48992	Aug 09 2007 12:30:42 CDT	Aug 10 2007 02:07:14 CDT
	870781280	871011216	229936	Aug 10 2007 06:41:06 CDT	Aug 12 2007 22:33:22 CDT
	871022304	871172336	150032	Aug 13 2007 01:38:10 CDT	Aug 14 2007 19:18:42 CDT
	871177984	871247520	69536	Aug 14 2007 20:52:50 CDT	Aug 15 2007 16:11:46 CDT
	871251808	871257360	5552	Aug 15 2007 17:23:14 CDT	Aug 15 2007 18:55:46 CDT
	871273088	871280880	7792	Aug 15 2007 23:17:54 CDT	Aug 16 2007 01:27:46 CDT
	871981200	871981328	128	Aug 24 2007 03:59:46 CDT	Aug 24 2007 04:01:54 CDT
	872003120	872114400	111280	Aug 24 2007 10:05:06 CDT	Aug 25 2007 16:59:46 CDT
	872636400	872655024	18624	Aug 31 2007 17:59:46 CDT	Aug 31 2007 23:10:10 CDT
	872659280	872661488	2208	Sep 01 2007 00:21:06 CDT	Sep 01 2007 00:57:54 CDT
	872665136	872693216	28080	Sep 01 2007 01:58:42 CDT	Sep 01 2007 09:46:42 CDT
	872698320	872731776	33456	Sep 01 2007 11:11:46 CDT	Sep 01 2007 20:29:22 CDT
	872741840	872787264	45424	Sep 01 2007 23:17:06 CDT	Sep 02 2007 11:54:10 CDT
	872788880	872794800	5920	Sep 02 2007 12:21:06 CDT	Sep 02 2007 13:59:46 CDT
	873212384	873269376	56992	Sep 07 2007 09:59:30 CDT	Sep 08 2007 01:49:22 CDT
	873271376	873278064	6688	Sep 08 2007 02:22:42 CDT	Sep 08 2007 04:14:10 CDT
	873284704	873425584	140880	Sep 08 2007 06:04:50 CDT	Sep 09 2007 21:12:50 CDT
	873431856	873617856	186000	Sep 09 2007 22:57:22 CDT	Sep 12 2007 02:37:22 CDT
	873649040	873668464	19424	Sep 12 2007 11:17:06 CDT	Sep 12 2007 16:40:50 CDT
	873671504	873676960	5456	Sep 12 2007 17:31:30 CDT	Sep 12 2007 19:02:26 CDT
	873700304	873715728	15424	Sep 13 2007 01:31:30 CDT	Sep 13 2007 05:48:34 CDT
	873718576	873830032	111456	Sep 13 2007 06:36:02 CDT	Sep 14 2007 13:33:38 CDT
	873831600	874030176	198576	Sep 14 2007 13:59:46 CDT	Sep 16 2007 21:09:22 CDT
	874034256	874094400	60144	Sep 16 2007 22:17:22 CDT	Sep 17 2007 14:59:46 CDT
	875260800	875263152	2352	Oct 01 2007 02:59:46 CDT	Oct 01 2007 03:38:58 CDT
	875264672	875279712	15040	Oct 01 2007 04:04:18 CDT	Oct 01 2007 08:14:58 CDT
	875282608	875283920	1312	Oct 01 2007 09:03:14 CDT	Oct 01 2007 09:25:06 CDT
	875286400	875286704	304	Oct 01 2007 10:06:26 CDT	Oct 01 2007 10:11:30 CDT
	875310352	875321088	10736	Oct 01 2007 16:45:38 CDT	Oct 01 2007 19:44:34 CDT
	875343488	875364768	21280	Oct 02 2007 01:57:54 CDT	Oct 02 2007 07:52:34 CDT
	875368464	875374176	5712	Oct 02 2007 08:54:10 CDT	Oct 02 2007 10:29:22 CDT
	875376032	875430880	54848	Oct 02 2007 11:00:18 CDT	Oct 03 2007 02:14:26 CDT
	875477232	875481312	4080	Oct 03 2007 15:06:58 CDT	Oct 03 2007 16:14:58 CDT
	875518592	875518720	128	Oct 04 2007 02:36:18 CDT	Oct 04 2007 02:38:26 CDT
	875522048	875533440	11392	Oct 04 2007 03:33:54 CDT	Oct 04 2007 06:43:46 CDT
	875535696	875539184	3488	Oct 04 2007 07:21:22 CDT	Oct 04 2007 08:19:30 CDT
	875558832	875562176	3344	Oct 04 2007 13:46:58 CDT	Oct 04 2007 14:42:42 CDT
	876126368	876145840	19472	Oct 11 2007 03:25:54 CDT	Oct 11 2007 08:50:26 CDT
	876163600	876171504	7904	Oct 11 2007 13:46:26 CDT	Oct 11 2007 15:58:10 CDT
	876176048	876176096	48	Oct 11 2007 17:13:54 CDT	Oct 11 2007 17:14:42 CDT
	876179648	876204112	24464	Oct 11 2007 18:13:54 CDT	Oct 12 2007 01:01:38 CDT
	876218256	876219920	1664	Oct 12 2007 04:57:22 CDT	Oct 12 2007 05:25:06 CDT
	876230416	876238384	7968	Oct 12 2007 08:20:02 CDT	Oct 12 2007 10:32:50 CDT
	876251104	876258016	6912	Oct 12 2007 14:04:50 CDT	Oct 12 2007 16:00:02 CDT
	876508656	876513216	4560	Oct 15 2007 13:37:22 CDT	Oct 15 2007 14:53:22 CDT
	876644432	876665376	20944	Oct 17 2007 03:20:18 CDT	Oct 17 2007 09:09:22 CDT
	876691056	876725552	34496	Oct 17 2007 16:17:22 CDT	Oct 18 2007 01:52:18 CDT
	876775760	876775808	48	Oct 18 2007 15:49:06 CDT	Oct 18 2007 15:49:54 CDT
	876778176	876778528	352	Oct 18 2007 16:29:22 CDT	Oct 18 2007 16:35:14 CDT
	876811680	876812880	1200	Oct 19 2007 01:47:46 CDT	Oct 19 2007 02:07:46 CDT
	876829952	876830288	336	Oct 19 2007 06:52:18 CDT	Oct 19 2007 06:57:54 CDT
	876848256	876852048	3792	Oct 19 2007 11:57:22 CDT	Oct 19 2007 13:00:34 CDT
	876907136	876908432	1296	Oct 20 2007 04:18:42 CDT	Oct 20 2007 04:40:18 CDT
	876911248	876912752	1504	Oct 20 2007 05:27:14 CDT	Oct 20 2007 05:52:18 CDT
	877094496	877095488	992	Oct 22 2007 08:21:22 CDT	Oct 22 2007 08:37:54 CDT
	877098416	877107712	9296	Oct 22 2007 09:26:42 CDT	Oct 22 2007 12:01:38 CDT
	877112112	877150816	38704	Oct 22 2007 13:14:58 CDT	Oct 23 2007 00:00:02 CDT
	877157088	877172864	15776	Oct 23 2007 01:44:34 CDT	Oct 23 2007 06:07:30 CDT
	877179216	877187360	8144	Oct 23 2007 07:53:22 CDT	Oct 23 2007 10:09:06 CDT
	877190640	877204800	14160	Oct 23 2007 11:03:46 CDT	Oct 23 2007 14:59:46 CDT