key issue: using `setattr` to change level names, keeps unwanted duplicates.
I am cleaning some data where I have sevearl factor levels, all of which are the same, appearing as two or more distinct levels.
(This error is due mostly to typos and file encoding issues)
I have 153K factors, and abot 5% need to be corrected.
**Example**
In the following example, the vector has three levels, two of which need to be collapsed into one.
incorrect <- factor(c("AOB", "QTX", "A_B")) # this is how the data were entered
correct <- factor(c("AOB", "QTX", "AOB")) # this is how the data *should* be
> incorrect
[1] AOB QTX A_B
Levels: A_B AOB QTX <~~ Note that "A_B" should be "AOB"
> correct
[1] AOB QTX AOB
Levels: AOB QTX
The vector is part of a `data.table`.
Everything works fine when using the `levels<-` function to change the level names.
However, if using `setattr`, then unwanted duplicates are preserved.
mydt1 <- data.table(id=1:3, incorrect, key="id")
mydt2 <- data.table(id=1:3, incorrect, key="id")
# assigning levels, duplicate levels are dropped
levels(mydt1$incorrect) <- gsub("_", "O", levels(mydt1$incorrect))
# using setattr, duplicate levels are not dropped
data.table::setattr(mydt2$incorrect, "levels", gsub("_", "O", levels(mydt2$incorrect)))
# RESULTS
# Assigning Levels # Using `setattr`
> mydt1$incorrect > mydt2$incorrect
[1] AOB QTX AOB [1] AOB QTX AOB
Levels: AOB QTX Levels: AOB AOB QTX <~~~ Notice the duplicate level
Any thoughts on why this is and/or any options to change this behavior? (ie `..., droplevels=TRUE` ?)
Thanks
--------------
solution
> mydt2$incorrect
# [1] AOB QTX AOB
# Levels: AOB AOB QTX
m3.drop <- copy(mydt2)
m3.fact <- copy(mydt2)
m3.fact[, incorrect:=factor(incorrect)]
m3.drop[, incorrect:=droplevels(incorrect)]
identical(m3.fact, m3.drop)
# [1] TRUE
Warning message:
In `levels<-`(`*tmp*`, value = if (nl == nL) as.character(labels) else paste0(labels, :
duplicated levels will not be allowed in factors anymore