Data Software
Resume:
Tie Liang
Guilderland, NY 12084
********@*****.***
EDUCATION
Research and Evaluation Methods Program
University of Massachusetts, Amherst, MA September, 2005-April,
2009
Degree: Ph. D
Concentration: Psychometrics and Statistics
PUBLICATIONS (published)
Liang, T., Han, K. T., & Hambleton, R. K. (2009). ResidPlots-2: Computer
software for IRT graphical residual analyses [computer program].
Applied Psychological Measurement, 33(5), 411-412.
Liang, T., & Wells, C. S. (2009) A model fit statistic for the generalized
partial credit model. Educational and Psychological Measurement,69(6),
913-928.
PUBLICATIONS (under Journal review)
Liang, T., & Wells, C. S. A nonparametric approach for assessing model fit
in a mixed format test.
PUBLICATIONS (under submission process)
Liang, T., & von Davier, A. A. Cross validation: An alternative
continuization method in kernel equating.
Liang, T., Wells, C.S., & Hambleton, R.K. An assessment of the
nonparametric approach for evaluating the fit of item response models.
Liang,T., Wells, C. S., Han, K. T., & Sireci,S.G.. Evaluating item
parameter drift in computerized adaptive testing.
CONFERENCE PRESENTATIONS
Liang, T., & von Davier, A. A. (2009). Cross validation: An alternative
continuization method in kernel equating. Paper presented at the
International Meeting of Psychometric Society, Cambridge, UK.
Liang, T., & Wells, C. S. (2008). A nonparametric approach for assessing
model fit in a mixed format test. Paper presented at the National
Council on Measurement in Education meeting, NY.
Liang, T., Han, K. T., & Baldwin, P. (2008). ResidPlots--a computer
software for graphical residual analysis. Poster presented at the
National Council on Measurement in Education meeting, NY.
Liang,T., Wells, C. S., Han, K. T., & Sireci,S.G. (2007). Evaluating item
parameter drift in computerized adaptive testing. Paper presented at
the meeting of the Northeastern Educational Research Association,
Rocky Hill, CT.
Liang, T., & Wells, C. S. (2007). A model fit statistic for the generalized
partial credit model. Paper presented at the meeting of the National
Council on Measurement in Education, Chicago, Illinois.
Wells, C. S., Hambleton, R. K., & Liang, T. (2008, July). A nonparametric
approach for investigating model fit in item response theory. Paper
presented at the III European Congress of Methodology, Oviedo, Spain.
WORK EXPERIENCE
Excelsior College, Albany, NY
Psychometrician September, 2009-Now
. Coordinate with Test Development Department to build test forms and
automate the process of form building.
. Provide training for test developers to understand the basics of
psychometrics.
. Supervise the data analysts' work and set up a quality control system to
double check results regarding data management and preliminary item
analyses.
. Create, modify and maintain databases for item bank and student
information system.
. Conduct and manage score testing activities using automated processes in
Access and investigate and resolve problems if there are any.
. Set up standard setting procedure and provide rater training.
. Determine the appropriate psychometric theory for evaluating exams and
document the procedures to implement psychometric methods for analyses.
Both Classical Test Theory and Item Response Theory are applied for
different tests in terms of data collection design, item analyses,
equating, linking and scaling, scoring, standard setting.
. Prepare and assist in publishing annual technical reports.
. Produce computer programs for data analysts and other psychometricians
to automate daily psychometric tasks and activities.
. Carry out research in measurement theory and practice to improve testing
procedures. Some ongoing topics are Differential Item Functioning, item
exposure control and speededness.
Center for Educational Assessment, Amherst, MA
Auditor Summer, 2008 and summer, 2009
Under supervision of Professor Hambleton, conducted equating for MCAS
tests to audit the results from Measured Progress. 20 tests covering Math,
Science and ELA were equated between the current year and the previous
year. The procedure of analyses includes computing p values, producing
delta plot and a and b plots, calibrating data using fixed common item
parameter (FCIP) method, building test characteristic curves, producing
conversion tables.
ACT, Inc., Iowa City, IA
Pychometric Intern June, 2008 - July,
2008
Worked on a project to investigate the performance of various equating
procedures for equating mixed format tests to MC only tests. The goal was
to find an appropriate equating procedure(s), which will produce comparable
scale scores between the mixed format form and the MC only form. 180
factors (3 ability distributions for new forms x 2 new form sample sizes x
10 equating methods with smoothing or scaling methods x 3 test formats)
were examined. The results were obtained and the paper is in preparation
for publication.
Center for Educational Assessment, Amherst, MA
Psychometric Intern June, 2007 - July,
2007
Evaluated the Massachusetts Adult Proficiency Test (MAPT) which is a
multi-stage computer adaptive test. Analyses included evaluation of
content validity, item analysis, assessment of IRT model fit, investigation
of differential item functioning, standard setting.
Harcourt Assessment, San Antonio, TX
Psychometric Intern June, 2006 - July,
2006
Programmed computer adaptive testing (CAT) procedure and produced a
computer software for Harcourt to implement CAT. The complete computer
package covers item selection, content balancing, item exposure control,
ability estimation.
RESEARCH EXPERIENCE
Research and Evaluation Methods Program, School of Education,
University of Massachusetts Amherst
Research Assistant September, 2005-April,
2009
. Applying IRT to the medical field
In collaboration with University of Massachusetts Medical School and
under supervision of Professor Hambleton, the IRT analyses were conducted
to develop measures of nicotine dependence. The process of analyses covered
the following steps: first, data cleaning and management. Second, checking
dimensionality using structural equation modeling. Third, computing point
biserial for each item and removing items with low point biserial. Fourth,
checking IRT model fit. Finally, linking between two groups of participants
to investigate differential item function (DIF) items. The paper will be
published in one of the medical journals.
. Proposing a model fit statistic for the generalized partial credit
model(GPCM)
This new fit statistic involves three major parts. First, finding a
nonparametrically estimated item response function (IRF) for partial credit
data using kernel smoothing (Fast Fourier Transformation method was adopted
to realize it). Second, finding the optimal GPCM item parameter values via
least square method using the kernel smoothed IRF. Lastly, using a
parametric bootstrapping method to approximate the significance level of
the difference between the nonparametric IRF and the parametric IRF
mentioned in the previous two steps. Under the guidance of Professor Craig
Wells, several studies were conducted to demonstrate that the proposed fit
statistic is promising in controlling type I error rate, providing adequate
power and giving graphical representation under various combinations of
sample sizes and test lengths. Associated with the research, a FORTRAN
program was completed for users who want to apply this fit statistic. The
paper was presented at the National Council on Measurement in Education
(NCME) meeting in 2007 and is published in Educational and Psychological
Measurement.
. Proposing a model fit statistic for a mixed format test
As an extension of the previous project, this one was to examine the
performance of the proposed fit statistic for a common testing scenario-a
mixed format test including three-parameter logistic model (3PLM), two-
parameter logistic model (2PLM), and generalized partial credit model
(GPCM). The results were very promising, especially compared to other fit
statistics. A FORTRAN program was prepared for users to apply this fit
statistic to a mixed format test which is popular with many state testing
programs. In the program, Fast Fourier Transformation was used to do kernel
smoothing. Nonlinear fitting, maximum likelihood and least square methods
were applied for estimating item parameters of each model, respectively.
The paper was presented at the National Council on Measurement in Education
(NCME) meeting in 2008 and is under journal review.
. Creating software--ResidPlots for graphical residual analysis (now
available at http://www.umass.edu/remp/software/residplots/)
Under the direction of Professor Ronald Hambleton, both the software and
the manual were completed. ResidPlots provides a powerful tool for
graphical residual analyses. The advantages of this software include
several notable features. (1) ResidPlots supports all of the most widely
used IRT models including three dichotomous models (1PLM, 2PLM, 3PLM) and
three polytomous models (GRM, GPCM, NRM). (2) ResidPlots provides
considerable flexibility with respect to the number and size of the
intervals for which the residuals are computed. (3) ResidPlots allows users
to decide what type of error bars they wish to have displayed. (4)
ResidPlots provides three sets of plots, first, at the item level, raw
residual plots with error bars and standardized residual plots, second, at
the test level, ResidPlots can show standardized residual distributions,
item fit plots and score fit plots from both empirical and simulated data,
third, observed score distribution and predicted score distribution are
produced too. In addition, some tables including basic item fit and person
fit statistics and summaries of residuals are provided.(5) The user-
friendly interface is convenient and straightforward. Users merely point to
the syntax file used to run PARSCALE or BILOG-MG or MULTILOG and ResidPlots
will provide the analysis. Additionally, users can access any plot by
pointing and clicking the options. The first version of the software was
introduced in detail at a National Council on Measurement in Education
(NCME) poster session in 2008. A write up on the extended software
developed over the last year with support from the College Board is now
published in Applied Psychological Measurement.
. Application of ResidPlots to College Board AP data
Under the direction of Professor Ron Hambleton, model data fit was
evaluated for Advanced Placement (AP) tests using ResidPlots. This
application demonstrated ResidPlots will be very convenient and flexible
for users who want to do graphical residual analysis.
. Evaluating Item Parameter Drift in Computerized Adaptive Testing
This project was under the direction of Professor Craig Wells and
Professor Steve Sireci. The purpose was to examine the effectiveness of two
statistics that could be used to detect item parameter drift (IPD) in the
multistage adaptive testing context--Lord's chi square (Lord, 1980) and the
standardization index (Dorans & Kulick, 1986). The effectiveness of these
statistics was evaluated using simulated data and real data from the
Massachusetts Adult Proficiency Test. Both FORTRAN and SPLUS code were
developed to get Type I error rate and power into the analyses. This paper
was presented at the Northeastern Educational Research Association (NERA)
in 2007.
. Proposing a new continuization procedure under the kernel equating
framework
This project was carried out in collaboration with Alina von Davier
from ETS. The newly proposed cross-validation method was compared with
kernel equating method and traditional equipercentile equating. 108
conditions were conducted in the study by varying three important factors
(sample size, test variable, method) in equating. The results indicated
that the cross validation procedure provided smaller bias and standard
error of equating than the other two equating methods. The beauty of the
new method is the poisson likelihood function established through cross
validation takes into account the poisson data variance while penalty
functions used in kernel equating procedure do not. The paper was presented
at the International Meeting of Psychometric Society in July, 2009.
. Collecting literature regarding comparability of computer based test
(CBT) and paper-and-pencil test (PPT)
Completed a literature review with Professor Steve Sireci on the topic
of comparability of CBT and PPT.
. Collecting literature regarding effects of reviewing and changing answers
on students' performances on computer-based testing and paper-pencil
testing
Compiled articles for Professor Steve Sireci for a final report to the
Department of Education in Massachusetts.
. Collecting literature on item exposure control methods in computer based
testing
With Professor Ron Hambleton's suggestions, prepared a bibliography to
summarize and criticize item exposure control methods proposed or used in
computer based testing.
. Collecting literature on IRT model fit statistics
Compiled articles and papers for Professor Ron Hambleton for a
bibliography on IRT model fit statistics. This work continues.
Student Development and Pupil Personnel Program, School of Education,
University of Massachusetts Amherst
Research Fellow September, 2004-August, 2005
My three main responsibilities were first, develop innovative program
models to improve teaching and learning or educational outcomes in schools,
second, help K-12 leaders and practitioners use research findings and
methods to improve school counseling practice by enabling them to set
standards, establish benchmarks, implement evidence-based interventions,
measure student learning outcomes, and evaluate program effectiveness.
Finally, coordinate between the program, schools and department of
education in Massachusetts on data collection, data analyses and annual
report preparation.
COMPUTER SKILLS
Statistical packages: S-PLUS, SPSS, LISREL, R, SAS
Measurement packages: MULTILOG, PARSCALE, BILOG-MG, TESTGRAF,
TESTFACT, DIMTEST
Programming languages: FORTRAN
Microsoft products: WORD, EXCEL, ACCESS
Others: LOTUS, ORACLE
MAJOR COURSEWORK
. Advances in Testing Methods
. Advanced Measurement Seminar
. Principles of Educational and Psychological Testing
. SAS programming
. Advances in Item Response Theory
. Applied Multivariate Analyses
. Validity Theory and Validity Applications
. Introductions to Item Response Theory
. Scaling Methods for the Behavioral Sciences
. Psychometric and Statistical Modeling
. Classical Test Models and Practices
. Educational Statistics
. Fundamentals of Test Construction
. Structural Equation Modeling
. Educational Research Methods
PROFESSIONAL ASSOCIATIONS
2005- present American Educational Research Association
2005- present National Council on Measurement in Education
2007- present Northeastern Educational Research Association
REFERENCES
Ronald K. Hambleton
Distinguished University Professor
University of Massachusetts-Amherst
Email: ***@****.*****.***
Stephen G. Sireci
Professor of Research and Evaluation Methods Program
University of Massachusetts-Amherst
Email: ******@****.*****.***
Craig S.Wells
Assistant Professor of Research and Evaluation Methods Program
University of Massachusetts-Amherst
Email: *******@****.*****.***
Contact this candidate