[Fabiola Lara]

The core IDELA tool does not include indicators on learning involvement or child-wellbeing, but we do try to address these important topics in a few different ways. The IDELA tool has optional items about children’s learning approaches (persistence, curiosity, attention, etc.). The items have not been validated as part of the core tool, but we are continuing to test and analyze their utility. In addition, we encourage teams to complement the IDELA tool with other measures that do measure these topics. For example, IDELA has been alongside classroom environment tools like the ECERS as well as child well-being measures that address children’s stress and psychosocial wellbeing.

[Fabiola Lara]

Yes, it is important that all enumerators are delivering the items to children in the same way to achieve standardization and ultimately, reliable data. Enumerators should memorize items for smooth delivery of assessment when assessing with young children.

[Jonathan Seiden]

Hi Sarah,

I’d recommend using KoBo over Tangerine for administering IDELA. The XLSForm that is available here is easily adaptable is already optimized for use in KoBo. We may be updating those templates to a more user-friendly version, but they work perfectly fine as-is. XLS forms make adding additional languages very easy as you just have to translate the “labels” for questions and choice answers.

It’s quite common to have the instructions in a colonial language and then the child-facing language in a local language. Just ensure you translate the words in **bold** into local language.

If you make a French version in XLSForm, we’d love for you to share the final version so that others can use it in the future.

[Frannie Noble]

This response comes from Lauren Pisani, an Advisor for Learning Research at Save the Children and an IDELA expert.

I suppose it could but this would be difficult to observe from a group of children. You would need highly trained observers and the process would benefit from individual interviews with each child. I don’t think you would get all that you want to know about children’s emotional wellbeing from a group observation.

[Frannie Noble]

And the person who originally posted the question responded,

Yes, thanks a lot for the sampling discussion. Yes, we too were thinking of doing a census. The purpose of the assessment is kind-of measurement of the effectiveness of the ECCD classes and school readiness of those children to enter the school in September 2018.
Considering the difficulty of gathering all parents in the villages (due to seasonal works in the fields, taking care for other children at home, etc.) we will reach out as much parents as possible to ensure at least the highest respondent rate closest to the possible rate the census could provide.

[Fabiola Lara]

Lauren Pisani, Research Advisor with Save the Children, replied:

With an impact evaluation, you are typically trying to determine whether an intervention has had a significant effect on the treatment group compared to some kind of control or comparison group. You need a sample size that is large enough to conduct the statistical tests to determine the magnitude of the difference between the two groups, a small group of children will not be enough to come to a strong conclusion. Also, if the intervention period is short, you may not find substantial changes in children’s skills over this short time. If this is the case and you have a short time period, you could use IDELA to get a general sense of the skills that children in your program possess but it will not be an impact evaluation.

• This reply was modified 5 years, 5 months ago by Fabiola Lara.

[Jonathan Seiden]

We have detailed instructions for how to clean and analyze IDELA data here.

In essence, each domain score (Emergent Numeracy, Emergent Literacy, Motor, and Social-Emotional) is calculated by the average percent correct for the tasks in each domain. So, in the Motor domain, if a child hopped 7/10 steps, drew 6/8 body parts, drew a good triangle with 3 sides and folded 1/4 folds, they would score 70% on Hopping, 75% on Drawing a Person, 100% on Drawing a Shape and 25% on Folding. The Motor score would be (70%+75%+100%+25%)/4= 67.5%. The Total IDELA score is simply the average for the four core domain scores.

The Excel Template also automatically will calculate the scores based on the default IDELA assessment. You can access it here.

[Jonathan Seiden]

We don’t have specific recommendations for technology for IDELA. You definitely do not need “top-shelf” devices like iPads to work. At the same time, it’s important to buy devices that will not break too quickly. If you have a complex form, newer tablets will typically run faster, so more RAM is generally helpful.

KoBo Toolbox (our recommended electronic data collection platform) has general guidelines on devices here: http://help.kobotoolbox.org/collecting-data/recommended-devices-for-data-collection

[Fabiola Lara]

Jonathan Seiden, Senior Research Specialist, replied:

There are definitely pluses and minuses that tablet administration has. On one hand, having the administration on the tablet makes it move a lot more quickly because there are fewer sheets lying around (just the tablet+stimulus cards and materials). On the other hand, it can be distracting for some children and there are technical glitches that happen. Overall, I think there are a lot more pluses than minuses, and we can really improve the reliability of data by ensuring we don’t get missing data (and that the data was actually collected as they said it was and not just filling out sheets under a tree somewhere).

[Fabiola Lara]

This not an uncommon question and challenge. Almost all schools or ECD groups have challenges with finding adequate spaces, especially quiet ones, for when assessing young children. A key tip would be to first make sure that there is someone on your team that can get a sense of the space (school, center, etc.) days prior to field testing in order to better understand how children and enumerators will be seated during the assessment. This person should also be present during field testing and assist enumerators on where to sit with children. Depending on the availability of space, children can sit with the enumerator under a tree, in the principal’s office, in a general meeting area for teachers, hallways, or any other extra areas that can offer some level of silence – even if complete silence is not guaranteed.

[Fabiola Lara]

Some things to consider would be to make sure that you have a realistic timeline in terms of when the review (adaptation and translation) of the tool, training of enumerators and data collection will need to take place. It is also important to take into consideration any known outside factors (school closures due holidays, beginning/end of term, etc.), intervention start dates, etc. In short, starting sooner rather than later is important.

[Nivida Chandra]

Hi,

In using the tablets, if we follow certain checklists and protocols, it can be a more reliable and accurate means of data collection.

We had our enumerators first train on paper, since they were not familiar with data collection on tablets. On paper, they learnt about IDELA. They learnt to administer IDELA on paper, and kept a copy with them just in case.
We then carved out about 2 days in the 5 day training for them the learn about tablets (switching them on/off, opening the app, using it online/offline, uploading, trouble shooting, batter conversation, not using the tablet for purposes other than data collection). We did a ton of field practice using the tablets.

The other thing we did was code only the question and response texts on the tablets. We kept all the visual items in a large kit – which each enumerator had to place in order before beginning any assessment. We also gave everyone a flip book which had only one question+image per sheet. All of this was to ease their use of materials and tablets together, which can otherwise be a source of confusion.

We also disallowed the skipping of items in the tablets. This immensely reduced the possibility of returning to questions or skipping them by mistake.

We also did daily checks – data had to be uploaded daily and we would check each evening if everyone’s data was in, and if they were missing data by any chance.

And of course, intense monitoring in the field. If we felt someone was not coping with the administration, they were immediately pulled out and trained some more.

After a couple of weeks, it becomes natural to use the tablets and in my personal opinion, it does not adversely affect the data collection at all.

[Nivida Chandra]

Hi,

Since there is no norm against which we compare the data, the questions of reliability and validity hold a different meaning in this context. I know Jonathan has responded to this in better detail, and I entirely agree with him.

I’d just like to add that one way of making sure that your adaptation is sound and evenly scored is inter-rater reliability. And to keep that going for as long as possible.
In using it in Delhi (India), we kept a close eye on the data and every enumerator was double scored 4-5 times. If the master trainers / supervisors felt they needed closer attention, then even more.

[Fabiola Lara]

Not all participants that attend an enumerator training are required to be selected for actual data collection. Generally, this is up to the discretion of program managers or staff leading data collection. Some partners opted for a larger training where they could then see how participants performed and then select the most competent enumerators for data collection. Others had already made careful selections prior to the training and therefore, everyone at the training was to be a part of data collection.

[Fabiola Lara]

To my knowledge, there have not been issues as such but only if the item is not adapted to what is relevant to the context. For partners, do you find that this is the case for you when you have used IDELA in the field?

[Author]

Replies