RESEARCH
Validation of the shake test for detecting freeze damage to adsorbed vaccines
Validation de l'épreuve d'agitation pour la détection des dommages occasionnés aux vaccins adsorbés
Validación de la prueba de agitación para detectar daños por congelación en las vacunas adsorbidas
Ümit KartogluI, *; Nejat Kenan ÖzgülerII; Lara J WolfsonIII; Wiesław KurzatkowskiIV
IDepartment of Immunization, Vaccines and Biologicals, World Health Organization, 20 Avenue Appia, 27 Geneva 1211 Switzerland
IIEmergency Department, Karaman Hayat Medical Centre, Karaman, Turkey
IIIGlobal Influenza Programme, World Health Organization, Geneva, Switzerland
IVNational Institute of Hygiene, Warsaw, Poland
ABSTRACT
OBJECTIVE: To determine the validity of the shake test for detecting freeze damage in aluminium-based, adsorbed, freeze-sensitive vaccines.
METHODS: A double-blind crossover design was used to compare the performance of the shake test conducted by trained health-care workers (HCWs) with that of phase contrast microscopy as a "gold standard". A total of 475 vials of 8 different types of World Health Organization prequalified freeze-sensitive vaccines from 10 different manufacturers were used. Vaccines were kept at 5 °C. Selected numbers of vials from each type were then exposed to "25 °C and "2 °C for 24-hour periods.
FINDINGS: There was complete concordance between HCWs and phase-contrast microscopy in identifying freeze-damaged vials and non-frozen samples. Non-frozen samples showed a fine-grain structure under phase contrast microscopy, but freeze-damaged samples showed large conglomerates of massed precipitates with amorphous, crystalline, solid and needle-like structures. Particles in the non-frozen samples measured from 1 μm (vaccines against diphtheria-tetanus-pertussis; Haemophilus influenzae type b; hepatitis B; diphtheria-tetanus-pertussis-hepatitis B) to 20 μm (diphtheria and tetanus vaccines, alone or in combination). By contrast, aggregates in the freeze-damaged samples measured up to 700 μm (diphtheria-tetanus-pertussis) and 350 μm on average.
CONCLUSION: The shake test had 100% sensitivity, 100% specificity and 100% positive predictive value in this study, which confirms its validity for detecting freeze damage to aluminium-based freeze-sensitive vaccines.
RÉSUMÉ
OBJECTIF: Déterminer la validité de l'épreuve d'agitation pour la détection de dommages occasionnés par la congélation aux vaccins adsorbés aluminiques sensibles à la congélation.
MÉTHODES: Un essai croisé en double aveugle a été utilisé pour comparer l'interprétation de l'épreuve d'agitation conduite par des professionnels de la santé formés à cet effet avec celle obtenue par microscopie en contraste de phase en tant qu'«étalon-or». Au total 475 flacons de 8 différents types de vaccins pré-qualifiés par l'OMS, sensibles à la congélation et provenant de 10 fabricants différents, ont été utilisés. Les vaccins ont été conservés à 5 °C. Un nombre choisi de flacons de chaque type a alors été exposé à une température comprise entre -25 °C et -2 °C pour des périodes de 24 heures.
RÉSULTATS: Il a été observé une totale concordance des résultats obtenus par les professionnels de la santé et ceux de la microscopie en contraste de phase dans l'identification des flacons endommagés par la congélation et des échantillons non congelés. Les échantillons non congelés présentaient une structure à grains fins en microscopie à contraste de phase, alors que les échantillons endommagés par la congélation présentaient de gros agglomérats de précipités accumulés présentant des structures amorphes, cristallines, solides et en aiguille. En ce qui concerne les échantillons non congelés, les particules mesuraient entre 1 μm (vaccins contre diphtérie-tétanos-coqueluche; Haemophilus influenzae type b; hépatite B; diphtérie-tétanos-coqueluche-hépatite B) et 20 μm (vaccins contre diphtérie et tétanos, simples ou combinés). Les agrégats des échantillons endommagés par congélation mesuraient quant à eux jusqu'à 700 μm (diphtérie-tétanos-coqueluche) et 350 μm en moyenne.
CONCLUSION: Dans cette étude, l'épreuve d'agitation du vaccin a présenté une sensibilité de 100 %, une spécificité de 100 % et une valeur prédictive positive de 100 %, ce qui confirme sa validité pour la détection de dommages occasionnés par la congélation aux vaccins adsorbés aluminiques sensibles à la congélation.
RESUMEN
OBJETIVOS: Determinar la validez de la prueba de agitación para la detección de daños por congelación en las vacunas adsorbidas en adyuvantes de aluminio sensibles a la congelación.
MÉTODOS: Se empleó un proyecto cruzado de doble ciego para comparar el rendimiento de la prueba de agitación realizada por personal sanitario formado con el de la microscopía de contraste de fases como «criterio de valoración de referencia». Se empleó un total de 475 viales de ocho tipos diferentes de vacunas sensibles a la congelación y precalificadas por la Organización Mundial de la Salud, procedentes de 10 fabricantes distintos. Las vacunas se mantuvieron a 5 º C. Un número determinado de viales de cada tipo se expuso a temperaturas de " 25 º C y " 2 º C durante períodos de 24 horas.
RESULTADOS: Se observó una coincidencia total entre el personal sanitario y la microscopía de contraste de fases a la hora de identificar los viales dañados por congelación y las muestras no congeladas. Las muestras no congeladas presentaron una estructura de grano fino al observarlas mediante un microscopio de contraste de fases, mientras que las muestras dañadas por congelación mostraron grandes conglomerados de precipitados concentrados con estructuras amorfas, cristalinas, sólidas y aciculares. Las partículas de las muestras no congeladas medían entre 1 μm (vacunas contra difteria-tétanos-tos ferina, Haemophilus influenzae tipo b; hepatitis B, difteria-tétanos-tos ferina-hepatitis B) y 20 μm (vacunas contra la difteria y el tétanos, aisladas o combinadas). Por el contrario, los agregados de las muestras dañadas por congelación medían hasta 700 μm (difteria-tétanos-tos ferina), con una media de 350 μm.
CONCLUSIÓN: El test de agitación de este estudio tuvo una sensibilidad del 100%, una especificidad del 100% y un valor predictivo positivo del 100%, lo que confirma su validez para detectar los daños por congelación de las vacunas con adyuvantes alumínicos sensibles a la congelación.
Introduction
Good temperature control during the storage and transport of vaccines is critical to ensure their potency and safety. Liquid formulations of aluminium-based vaccines against diphtheria, pertussis, tetanus, hepatitis B and Haemophilus influenzae type b, alone or in combination (adsorbed vaccines), should not be frozen.1 However, practices that expose vaccines to sub-zero temperatures are widespread in both developed and developing countries at all health system levels.2-11 The most recent systematic literature review of vaccine freezing practices showed that accidental freezing occurs across all parts of the cold chain.12 Between 14% and 35% of refrigerators or transport shipments were found to have exposed vaccines to freezing temperatures, while in studies that examined all segments of the distribution chain, between 75% and 100% of the vaccine shipments were exposed. More rigorous study designs were associated with higher levels of exposure to freezing.
When a vaccine is damaged by freezing, the potency lost can never be restored - the damage is permanent.13-16 Freeze-damaged vaccines have lower immunogenicity and are more likely to cause local reactions, such as sterile abscesses.1,17
The shake test is designed to determine whether adsorbed vaccines have been affected by freezing. After freezing, the lattice (made up of bonds between the adsorbent and the antigen) in a vaccine is broken. Separated adsorbent tends to form larger, heavier granules that gradually settle at the bottom of the vial when this is shaken. When freezing and thawing cycles are repeated, the granules appear to increase in size and weight. In a typical demonstration of the shake test, two identical vials of a vaccine (i.e. from the same batch and the same manufacturer) that is suspected of having been exposed to freezing temperatures are selected; one of the two vials is purposely frozen and then thawed as the negative control, while the second vial serves as the vial to be "tested" against this negative control. The two vials are held together in one hand and shaken; they are then placed side by side on a flat surface. Provided the test vial has not been frozen, sedimentation is slower in the test vial than in the control vial that has been frozen and thawed.1,18 If the test vial has been frozen, the test and control vials will have similar sedimentation rates. Fig. 1 illustrates how the appearance of frozen (i.e. frozen and thawed, and therefore freeze-damaged) and non-frozen test vials compares to that of their frozen control vial, 1 minute and 28 seconds after shaking.
During the late 1980s, a shake test protocol was developed based on empiric observations in the field. However, its description was available nowhere except on a poster in the archives of the World Health Organization (WHO). Although the shake test is widely practiced in the field by staff at all levels of the health system, it has never been validated as a reference test by comparison to a "gold standard". The test is also used as a decision tool in accepting international shipments of freeze-sensitive vaccines if a temperature monitoring device indicates that freezing has occurred (i.e. a "freeze alarm" is activated if the temperature is continuously at or below "0.5 °C for more than 1 hour), and in determining whether vaccines exposed to temperatures below zero could be safely used. There are anecdotal reports of concerns from health-care workers (HCWs) about the usefulness of the shake test.
The most appropriate gold standard for the shake test (apart from testing the vaccine in humans, which is impractical) is visual observation under a phase contrast microscope. Phase contrast microscopy is a confirmed method of identifying freeze damage in vaccines. Damaged vaccines contain large conglomerates (massed precipitates with amorphous, crystalline, solid and needle-like structures), whereas vaccines maintained within the optimal temperature range (2 to 8 °C) show a fine-grain structure under phase contrast microscopy.19 These findings have been confirmed by scanning electron microscopy and X-ray analysis, which have shown that aluminium is the main element in the conglomerates (data not shown).
This study was designed to establish the shake test's sensitivity (proportion of vials identified as freeze-damaged by the shake test among vials identified as freeze-damaged by phase contrast microscopy) and specificity (proportion of vials identified as non-frozen by the shake test among vials identified as non-frozen by phase contrast microscopy), using phase contrast microscopy as the gold standard. Positive predictive value - i.e. the probability that a vial identified as freeze-damaged by the shake test is truly freeze damaged - was also calculated.
Methods
Sample selection
At the time of the study, 14 manufacturers had 8 freeze-sensitive products that were prequalified by WHO (details of prequalification are available at: http://www.who.int/immunization_standards/vaccine_quality/pq_system/en/index.html).
This study was designed to determine whether the sensitivity or specificity of the shake test varied depending on the specific product or manufacturer. To ensure that the study was statistically sound, it included all products produced by only one manufacturer (e.g. liquid Haemophilus influenzae type b vaccine produced by Merck, Whitehouse Station, USA); all manufacturers having only one product (e.g. hepatitis B vaccine produced by CIGB [Centro de Ingeniería Genética y Biotecnología], Havana, Cuba); a maximum of three randomly selected combinations of product and manufacturer (product selection per manufacturer was limited to a maximum of three; thus, all products from manufacturers with two or three products were included); a minimum of 30 vials of each vaccine type (reduced to 20 vials if more than one manufacturer produced the same product type); and 10-dose presentations (i.e. the most common type), unless the vaccine was only available in smaller doses (WHO experience in the field suggested that if the shake test worked with the selected 10-dose vial, it would also work with vials of any other size; also, vials with different doses have identical substances and mix, so that the behaviour of the shake test should be the same). Sample size calculations were based on an expected 95% specificity and 95% sensitivity of the shake test when compared with phase-contrast microscopy. The overall desired sample size was calculated to be 480 vials. The desired sample size by vaccine type was 30-140 vials, and by manufacturer, 30-60 vials. Table 1 illustrates the sampling framework for the study.
The sample sizes selected were appropriate to calculate specificity and sensitivity with a high degree of precision. A sample size of 480 could be considered unnecessarily high, but 30 samples of each vaccine type were needed to demonstrate the absence of a statistically significant difference between presentations.
Receiving and storing samples
Each vaccine manufacturer was asked to send either 25 or 35 vials of selected vaccines to WHO in Geneva, Switzerland. The five extra vials (practice vials) were needed for teaching the shake test, carrying out the interobserver variation test and validating the test protocol with phase contrast microscopy.
All vaccines were sent in insulated shipping containers with cool water packs (to maintain a temperature of 2 °C to 8 °C). Temperatures during shipment were monitored with the WHO prequalified 10-day electronic shipping indicator, Q-tag2plus® (Berlinger & Co. AG, Ganterschwil, Switzerland). All samples included in the study were received in good condition as indicated by the Q-tag2plus® and were then stored in a WHO 5 °C storage facility.
Preparation of samples
Samples were divided into three groups for preparing frozen and non-frozen vaccines. The temperature treatment plan for samples is shown in Table 2. First, the original label was removed from each vial and replaced with a study label having a 7-digit numerical code. The codes were assigned by the first co-investigator and were known only to that person until the study was completed. Samples to be exposed to negative temperatures were taken to the Thermometry and Ionizing Radiation Section of the Federal Office of Metrology in Berne, Switzerland. They were placed in one of two chambers in which temperature ranges were recorded as "25.1 °C to "24.7 °C, and "2.3 °C to "1.7 °C. After 24 hours, vaccines were removed from the temperature chambers and the physical status of each vial was examined. Vaccines were then transported to the National Institute of Hygiene in Warsaw, Poland, in insulated containers with cool water packs and one Q-tag2plus® per carton. Vaccines arrived at the study site within 23 hours of pickup and were immediately placed in a storage facility at 5 °C at the National Institute of Hygiene.
Three blinds
The study was organized to be fully blind in each of the three phases - shake test by HCWs, phase contrast microscopy by study centre staff and statistical analysis by the second co-investigator. These individuals had no information on the findings of the others and were unaware of the meaning of the coding in the data sheets.
Shake test by HCWs -first blinding
Five HCWs with no previous experience of vaccines or the shake test were recruited. The principal investigator taught the HCWs how to conduct the shake test following the standard "Shake test learning guide" (Appendix A, available at: http://www.who.int/vaccines-documents/DocsPDF06/847.pdf, pages 59-62). HCWs were given the extra five vials from each type of vaccine to practice with for half a day on their own. As a first step, interobserver variation was checked on 10 frozen and 10 non-frozen samples. During the interobserver variation tests, all HCWs performed "pass" and "fail" tests correctly, and all five were recruited.
Study vials were distributed to the HCWs with a freeze-damaged control vial for each vaccine type used. HCWs recorded the results of each test using an established code for "pass" and "fail" and the time taken to reach a decision. If a test vial contents sedimented at a similar or a faster rate than the contents of the frozen control vial, this was recorded as a "fail"; if the vial contents sedimented at a slower rate than the contents of the frozen control vial, this was recorded as a "pass".
Phase contrast microscopy -second blinding
Phase contrast microscopy was validated using the extra five vials from each type of vaccine. Once the procedure had been validated, all study samples were examined under phase contrast microscopy. Each vial was vigorously shaken, the aluminium crimping and the rubber stopper were removed from each vial, 10 μl of the vaccine in each vial were dropped onto a slide using a Biohit 0.5-10 μl automatic pipette (Biohyt Oyj, Helsinki, Finland), and a coverslip was placed over the sample. All samples were examined for structural formations under 200 × magnification and were photographed under 50 × magnification. The tests were conducted using a Docuval phase contrast microscope and Docuval and Planachromat 20/0.40 and 160/017 camera equipment (Carl Zeiss, Jena, Germany), and 24 × 36 mm Kodak 200 ASA negative film (Kodak, Hemel Hempstead, England). All photographs were digitized and particle size was measured. Results were coded numerically for "frozen" and "non-frozen" vaccines.
Statistical analysis -third blinding
Results from the shake test and phase contrast microscopy were tabulated separately by the second co-investigator on a 2 × 2 table for each product, by vaccine type, vaccine manufacturer, aluminium content and expiry date as well as together (phase contrast microscopy versus shake test). Sensitivity, specificity and positive predictive values were calculated.
Results
The study was conducted with 480 vials of 8 types of freeze-sensitive WHO prequalified vaccines from 10 manufacturers. During the unpacking of vaccines in Warsaw, 5 vials were broken and were excluded from the study. This reduced the sample to 475 vials. All vaccines exposed to "2 °C for 24 hours (117 vials, excluding one vial broken on arrival in Warsaw) were in a liquid state, as were vaccines kept at 5 °C. A total of 319 vials were not frozen and 156 were frozen.
Phase-contrast microscopy confirmed the known status of the vials: 319 were identified as non-frozen and 156 as frozen. Fig. 2 shows the appearance of non-frozen samples (fine-grain structure) and freeze-damaged samples (large conglomerates of massed precipitates with variable structures) under the phase contrast microscope.
Particles in samples identified as non-frozen measured between 1 μm (for vaccines for diphtheria-tetanus-pertussis; Haemophilus influenzae type b; hepatitis B; diphtheria-tetanus-pertussis-hepatitis B) and 20 μm (for vaccines against diphtheria and tetanus, alone or in combination). Aggregates in samples identified as frozen measured up to 700 μm (diphtheria-tetanus-pertussis) and 350 μm on average. The shape of aggregates in frozen samples varied; it included amorphous, solid, crystalline and needle-like structures with sharp edges.
Table 3 shows how the shake test results compared with those of phase contrast microscopy. There was complete concordance between the shake test results as interpreted by the HCWs and phase contrast microscopy readings, with no false positive or false negative readings. Sensitivity, specificity and positive predictive value were calculated as 100% each. Thus, the shake test correctly identified that a vaccine had been affected by freezing 100% of the time (95% confidence interval, CI: 0.97-1.00) and it also correctly identified that a vaccine has not been frozen 100% of the time (95% CI: 0.99-1.00).
Since the specificity, sensitivity and positive predictive value of the shake test were all calculated as being 100%, no further statistical analyses were conducted by manufacturer or product. The results fully support the original hypothesis that the sensitivity and specificity of the shake test do not vary by product, type of vaccine, vaccine manufacturer, aluminium content or expiry date.
Additional findings
An additional finding of the study was the time taken by HCWs to reach a conclusion. Although the time required is influenced by the experience of the HCW, results suggest that the test takes longer for smaller vials. The shortest decision time was 44 seconds with a 10-dose tetanus toxoid vaccine, and the longest was 20 minutes with a monodose Haemophilus influenzae type b vaccine. Apart from these extreme values, all other products were analysed within 1 to 5 minutes.
Since the "2 °C exposure did not produce any partial freezing (incomplete crystallization), the study team designed additional tests to complement the findings by generating slushy frozen vaccines. A total of 30 vials from 3 different manufacturers containing 7 different types of vaccines were exposed to "10 °C, with 15 minutes checks to record their freezing status. Eighteen of these vials were removed when they had reached a slushy but not fully solid frozen state, and 12 were allowed to freeze fully and reach a solid state. All samples were tested by the same HCWs in Warsaw using 7 different control vials and were examined by phase contrast microscopy, together with 7 matching vaccine vials kept at 5 °C. All vials that were slushy frozen and vaccines kept at 5 °C produced a "pass" test, and a fine-grain structure was observed in phase contrast microscopy. All 12 vials that were fully frozen produced a "fail" test, and large conglomerates were observed in phase contrast microscopy (Table 4 and Fig. 3). Sensitivity and specificity of the shake test for slushy vaccines were both calculated as 100% (sensitivity 95% CI: 0.86-1.00; specificity 95% CI: 0.93-1.00).
Discussion
This study was conducted to establish the sensitivity and specificity of the shake test by comparison against the actual freezing status of freeze-sensitive vaccines, using phase contrast microscopy as a gold standard. The concordance in establishing the status of a vaccine as frozen or non-frozen was 100% between the phase contrast microscopy and the shake test performed by HCWs. These findings indicate that the shake test has 100% sensitivity, specificity and positive predictive value.
Under the phase contrast microscope, frozen vaccines showed large conglomerates of large precipitates with variable structures. This confirms that freezing breaks the lattice between the adsorbent and the antigen, leading the aluminium to form granules that grow in size. Heavy granules sediment at a faster rate than lighter granules; this is the basis of the shake test.
None of the vaccines that were exposed to "2 °C for 24 hours were found to be frozen; all were in a liquid state. This confirms that actual freezing depends on various factors, including low temperature, duration of exposure to low temperature and agitation during the exposure. Under the phase contrast microscope, vaccines exposed to "2 °C looked identical to those kept at optimum temperatures - all showed fine-grain structure. In a temperature-monitoring study conducted in Thailand, investigators also found "pass" shake test results with vaccines exposed to negative temperatures documented by freeze indicators.20 This finding again confirms that exposure to negative temperatures and actual freezing are two different concepts. Since freeze indicators are the only practical tools available for checking or documenting whether vaccines have been exposed to negative temperatures, the authors strongly recommend the continued use of freeze indicators during in-country vaccine distribution of freeze-sensitive vaccines.
The shake test can also be used on slushy, partially frozen vaccines, and our results indicated that the lattice structure is broken only when solid freezing occurs.
We identified two publications claiming that the shake test is either impractical or not valid for identifying freeze-damaged vaccines. In a paper from Canada, based on a study of 80 vials of diphtheria-tetanus-pertussis and diphtheria-tetanus-pertussis-poliomyelitis vaccines, none of the frozen vials produced a positive shake test.15 The authors indicated that accelerated sedimentation was evident in all frozen vials but found that it took too long - up to 45 minutes - to produce a definitive result. Despite this finding, it is not clear how the authors concluded that the test is "impractical". An article from India was described as a "validation" study.21 However, the methods used did not correspond to a validation study design. Shake test results were not compared against any gold standard, no control vials were prepared and no standard shake test protocol was followed. Each participant was given one previously frozen and one never frozen vial, and was asked to report results after 15 minutes - an insufficient and arbitrary time-limit. Thus, this publication cannot be considered as a valid study of the shake test.
Two other publications support the shake test. One study evaluated Diphtheria-tetanus-pertussis and DT vaccines from six manufacturers, and concluded that the shake test is useful for absorbed vaccines.13 The other study did not examine the shake test directly, but documented a high rate of sedimentation in aluminium adjuvant vaccines kept at "18 °C compared to non-frozen vaccine samples; the sedimentation rate in frozen samples was 100% in 15-20 minutes compared to a maximum of 34% sedimentation in 24 hours for non-frozen samples.19
Our findings confirmed the importance of using a standard learning guide for training, coupled with a demonstration and coaching for HCWs with no prior knowledge and experience with the test. This approach resulted in HCWs being able to perform and read test results with 100% accuracy. The key to deciding whether a shake test has passed or failed is the patience of the observer. All HCWs were told that they should continue to observe until they were completely confident about the difference or similarity between the control and test vials. Accurately observing the sedimentation requires greater attention with smaller vials because the amount of liquid (and thus the height of the liquid in the vial) is significantly less in monodose and other small vials when compared to multidose vials.
These findings confirm the value of the shake test in deciding whether aluminium-based freeze-sensitive vaccines have been affected by freezing. The specificity and sensitivity of 100% found in this study will bring confidence to staff handling vaccines at the country level.
Note: The following videos about the shake test are available: Shake and Tell - video article (duration 00:22:17), available at: http://vimeo.com/8381355 - and Step by Step Shake Test - educational/instructional video (duration: 00:10:07), available at: http://vimeo.com/8389435
Acknowledgements
We thank the DHL-UNICEF Partnership for co-funding the validation study. We also thank WHO prequalified vaccine manufacturers BB-NCIPD Ltd, Bulgaria; Berna Biotech Korea Corp., Republic of Korea; CIGB, Cuba; GSK Biologicals, Belgium; Merck & Co., Inc., USA; Panacea Biotec Ltd, India; PT BioFarma, Indonesia; Serum Institute of India Ltd, India; Shantha Biotechnics Ltd, India; and Sanofi Pasteur, France, for participating in the study; Berlinger & Co. AG for donation of Q-tag2plus® devices; and Fride Martin, Steve Wiersma and Philippe Duclos for their assistance during development of the study protocol. We also thank Andrew Garnett for his assistance during the development of this article. Authors acknowledge Nigel Cryer (former DHL, United Kingdom) for establishing the link between the WHO and the DHL-UNICEF Partnership.
Funding: This study was co-sponsored by the DHL-UNICEF Partnership.
Competing interests: None declared.
References
1. Milstien J, Kartoglu U, Zaffran M. Temperature sensitivity of vaccines. Geneva: World Health Organization; 2006 (unpublished document WHO/IVB/06.10)
2. Burgess MA, McIntyre PB. Vaccines and the cold chain: is it too hot...or too cold? Med J Aust 1999;171:82. PMID:10474582
3. Milhomme P. Cold chain study: danger of freezing vaccines. Can Commun Dis Rep 1993;19:33-8. PMID:8467304
4. Lugosi L, Battersby A. Transport and storage of vaccines in Hungary: the first cold chain monitor study in Europe. Bull World Health Organ 1990;68:431-9. PMID:2208556
5. Hanjeet K, Lye MS, Sinniah M, Schnur A. Evaluation of cold chain monitoring in Kelantan, Malaysia. Bull World Health Organ 1996;74:391-7. PMID:8823961
6. Jeremijenko A, Kelly H, Sibthorpe B, Attewell R. Improving vaccine storage in general practice refrigerators. BMJ 1996;312:1651-2. PMID:8664720
7. Thakker Y, Woods S. Storage of vaccines in the community: weak link in the cold chain? BMJ 1992;304:756-8. doi:10.1136/bmj.304.6829.756 PMID:1571683
8. Wawryk A, Mavromatis C, Gold M. Electronic monitoring of vaccine cold chain in a metropolitan area. BMJ 1997;315:518. PMID:9329306
9. Bishai DM, Bhatt S, Miller LT, Hayden GF. Vaccine storage practices in pediatric offices. Pediatrics 1992;89:193-6. PMID:1734382
10. Bell KN, Hogue CJ, Manning C, Kendal AP. Risk factors for improper vaccine storage and handling in private provider offices. Pediatrics 2001;107:e100. doi:10.1542/peds.107.6.e100 PMID:11389298
11. Wirkas T, Toikilik S, Miller N, Morgan C, Clements CJ. A vaccine cold chain freezing study in PNG highlights technology needs for hot climate countries. Vaccine 2007;25:691-7. doi:10.1016/j.vaccine.2006.08.028 PMID:16968657
12. Matthias DM, Robertson J, Garrison MM, Newland S, Nelson C. Freezing temperatures in the vaccine cold chain: a systematic literature review. Vaccine 2007;25:3980-6. doi:10.1016/j.vaccine.2007.02.052 PMID:17382434
13. World Health Organization. The effects of freezing on the appearance, potency, and toxicity of adsorbed and unadsorbed DTP vaccines. Wkly Epidemiol Rec 1980;55:385-92.
14. Klotz SA, Normand R, Silberman R. Hepatitis B vaccine in healthy hospital employees. Infect Control 1986;7:365-9. PMID:3637168
15. Dimayuga R, Scheifele D, Bell A. Effects of freezing on DTP and DTP-IPV vaccines, adsorbed. Can Commun Dis Rep 1995;21:101-3. PMID:7647743
16. Dietz V, Galazka A, van Loon F, Cochi S. Factors affecting the immunogenicity and potency of tetanus toxoid: implications for the elimination of neonatal and non-neonatal tetanus as public health problems. Bull World Health Organ 1997;75:81-93. PMID:9141753
17. Mansoor O, Pillans PI. Vaccine adverse events reported in New Zealand 1990-95. N Z Med J 1997;110:270-2. PMID:9269289
18. World Health Organization. Guideline for establishing or improving primary and intermediate vaccine stores. Geneva: WHO; 2002 (WHO/V&B/02.34).
19. Aleksandrowicz J, Drozdz M, Fiejka M, Kurzatkowski W. Evaluation of the physico-chemical state of aluminium hydroxide in biopreparations stored in various conditions [in Polish]Exp Med Microbiol 1990;42:163-70.
20. Techathawat S, Varinsathien P, Rasdjarmrearnsook A, Tharmaphornpilas P. Exposure to heat and freezing in the vaccine cold chain in Thailand. Vaccine 2007;25:1328-33. doi:10.1016/j.vaccine.2006.09.092 PMID:17157419
21. Grover A, Amarjeet S, Aggarwal AK, Thakur JS. Does Indian version shake test correctly identify frozen damage vaccines? Bull Postgrad Inst Med Educ Res Chandigarh 2004;38:125-30.
(Submitted: 14 July 2009 - Revised version received: 23 November 2009 - Accepted: 1 December 2009 - Published online: 9 February 2010 )
* Correspondence to Ümit Kartoglu (e-mail: kartogluu@who.int).