We assess the quality of tests that are implemented and carry out analysis of tests for the various organizations involved in testing, such as qualifying examination bodies, educational institutions, and education services companies to ensure that the abilities of examinees are understood correctly.
- Is the difficulty of the questions within an appropriate range? Is the number of questions appropriate?
- Do the choices function well? Does each question distinguish between examinees with high level and low level ability?
- Is the pass line setting and method of dividing levels appropriate?
- Is it possible to make comparisons with previous test scores and ascertain changes in continuous test scores?
- What is the relationship between examinee attributes, grouping and test scores?
Item analysis - Analysis using classical test theory
| Output index name | Explanation of the index |
|---|---|
| Percentage of correct answers | Difficulty of questions in the test population |
| Point biserial correlation | How well does a question item discriminate between high and low abilities? |
| Biserial correlation | The correlation when the questions and test scores are assumed to follow a bivariate normal distribution |
| Choice selection rate | Choice selection status, function status |
| Actual choice rate | The actual number of choices as seen from the test results |
| Fundamental statistics | Fundamental statistical information about the test |
| Reliability factor | An index of the reliability of the test score |
| Standard error | The standard deviation for score assuming a certain examinee takes the test repeatedly |
| GP analysis table | A table and graph showing how the choices function for each level |
Analysis using IRT
| Output index name | Explanation of the index |
|---|---|
| Parameter a | An index of the sensitivity with which ability groups around value b are identified |
| Parameter b | The difficulty of a particular question |
| Parameter c | An index of the possibility of guessing the correct answer |
| Standard error a | The standard error for value a assuming repeated data acquisition and calculation |
| Standard error b | The standard error for value b assuming repeated data acquisition and calculation |
| Standard error c | The standard error for value c assuming repeated data acquisition and calculation |
| chi-square | The deviation between the percentage of correct answers obtained from the model and the percentage of correct answers obtained from the data |
| df | Degree of freedom (the number of division categories for ability score used to calculate the χ squared value) |
| p | The occurrence ratio of relevant data assuming equivalence between the model and the data |
| Fundamental statistics | Average, median, standard deviation, variance, range, minimum, maximum, sample size |
| Test characteristic curve | A graph of the correspondence between ability score and test score |
| Test information curve | A diagram showing the reliability of each ability score point |
Analysis using IRT - Equating -
| Output index name | Explanation of the index |
|---|---|
| Parameters after equating | Parameters a, b and c converted to the same scale |
| Test characteristic curve | A diagram showing the number of expected correct answers in the test score for each ability score after equating |
| Test information curve | A diagram showing the test information content in the test score for each ability score after equating |
- *Required data: A minimum of about 20% of questions must be anchor items for equating
(Anchor item: Questions that are the same across separately implemented test sets)
| Analysis of a mock examination for the National Medical Practitioners Qualifying Examination |
|
|---|---|
| Analysis of a proficiency test concerning asset management |
|
| A new type of recruiting test |
|
| Medical CBT |
|
| Keio University |
|
| Kumamoto University |
|
| Waseda University |
|
We offer consulting for clients who want to establish a test such as establishing qualification testing or computerizing an existing test.
Establishing a new qualification testing
- Sample test design
- Item pool design and application
- Designing and building tests
- Consulting regarding test implementation
Improving the quality of an existing qualification test
- Analysis of the quality of existing test questions
- Equating the difficulty and cutoff line of an existing test
- Consulting regarding methods of preparing test questions (distractors etc.)
- Consulting regarding how to provide feedback after testing
Computerization (CBT) of an existing test
- Designing and building CBT tests
- Sample test design in CBT development Sample test design in CBT development
- Item pool design and application in CBT building
- Provision of CAT (adaptive test) modules
- *CAT modules are currently patent pending.
Waseda University International Corporation
Waseda University International Corporation is currently running Tutorial English, a program for students to improve their communication in English. This program is offered by the Waseda University Open Education Center as a common subject for all undergraduate schools. Focusing on intensive practice in speaking to develop the ability to hold discussions in English, tutors teach small groups organized according to their level. JIEM helped to develop the WeTEC English placement test, and WeTECmini, a test to determine students’ levels before registering for the subject. We were asked to solve the following problems.
- The test for determining the level of examinees before registration had to be implement for a large number of people in a short time, as quickly and as accurately as possible
- The English placement test had to be highly reliable in order to serve as an index in organizing classes (four people) optimally by level, which is linked directly to educational effectiveness
Since Tutorial English counts towards credits, the number of students taking the course is increasing each year. A paper test involving the burden and time of implementation and scoring could not physically be processed to meet the conditions above. This realization led to the development of a new test. In order to meet these requirements, JIEM proposed the following;
- Use computer based testing (CBT) to simplify implementation of the test and scoring
- Use adaptive testing (CAT) to ensure highly reliable assessment in a short time
Work started at the start of summer in 2003. Professor Michiko Nakano, Head of the English Department at Waseda University Open Education Center prepared the test items, and a sample test was carried out. JIEM performed item analysis and IRT analysis on the sample test and set the parameters for the various test items. We then developed a CAT system incorporating these items and delivered it to the server at Waseda University.
Around ten thousand students have taken the test since the spring of 2004. The test is now managed by Waseda University International. (Number of examinees as of September 2004: 8,600)
Related URL：http://www.wui.co.jp/
Toppan Printing
The E-Learning Division of Toppan Printing Co., Ltd. has operated its mobile phone service helping students prep for the Eiken English test since March 2004. Service started on i-mode and EZ-web in March 2004 and on Vodafone from July, offering tests, learning and information for middle and high-school students at 315 yen a month.
The service uses CAT (adaptive testing) for the test part, enabling accurate assessment of English proficiency in a short time. The results allow students to focus on the English words and phrases they need to learn for efficient memorization. Since the service is phone-based, it can be used for study anywhere at any time.
Toppan Printing developed and operates this service, but JIEM provides the CAT module which is the engine for the test system, as well as the test questions and study content.
Features of the service
1. Users take English proficiency assessment tests and repetition training with the memorization tool which presents the number of words and phrases required to pass, enabling students to remember phrases using a consistent system.
- Focusing on words and phrases to learn using English proficiency assessment tests
- Development of a memorization tool
2. Menus which provide continuous motivation to learn
- Ranking: Weekly ranking of the number of words learned and ranking by Eiken grade from the results of the English proficiency assessment test results
- Eiken English-Japanese Dictionary DX: An English-Japanese dictionary enabling quick searches for the meaning of words that appear in the Eiken test. Showing frequency of appearance in Eiken
- Today’s question email magazine: A simple English question from each Eiken grade sent every day
- Wallpaper changer: Displays a selection of hard-to-remember words as wallpaper
Fuji Xerox Learning Institute Inc.
Fuji Xerox Learning Institute offers capacity development and aptitude testing for companies and other organizations. JIEM provides help with the following two areas.
(1) English proficiency development
Fuji Xerox Learning Institute receives contracts from client companies to develop and manage English proficiency measurement programs that are linked directly to employees’ work. Currently, the program is being run with existing tests and speaking tests developed in-house, which were developed with consulting from JIEM in methodologies for English proficiency measurement, analysis of sample tests and so on.
Fuji Xerox Learning Institute’s corporate clients introduced this program to each division and started testing in April 2004. JIEM’s CASEC is used for part of this program.
(2) Career Competency Business Office
The Career Competency Business Office offers consulting in the field of career development with a focus on competency. JIEM offers advice on verification of survey results and survey methods concerning the various surveys carried out as part of our consulting for client companies.
Related URL：https://www.fxli.co.jp/
Hyogo Prefectural Board of Education
In March 2004, the Hyogo Prefectural Board of Education carried out a learning survey of around 25,000 pupils at the 300 or so elementary and junior high schools in the prefecture. Students, parents and teaches answered a questionnaire about five subjects, Japanese and mathematics at elementary school and Japanese, mathematics, and English at junior high school. The aim was to evaluate the state of education in the prefecture, and to identify issues with teaching methods and consider improvements to ensure effective learning of the basics.
JIEM was chosen to conduct the learning survey, and we carried out the following tasks:
- Preparation of paper tests and questionnaire materials (printing, binding, and preparation of CDs)
- Mailing and collection of survey materials for the schools involved
- Scanning of mark sheets and scoring of written tests
- Analysis of score results and questionnaire data from students, parents and teachers
Example: Simple analysis, cross analysis, correlation analysis etc. - Printing and binding of the report
