MSSQL - Using REPLACE to Change Multiple Values To Single Value

I need a query to convert multiple values in one column to a single value.  Basically a Y/N situation.

As an example: If the field contains values 1, 2, or 3, return "Y", if the field contains values, 4, 5, or 6 return "N".

I know I can do a stack of replace statements, but is it possible to do a replace statement kind-of like an "in" clause...like

replace(replace(field,['1','2',['3'], 'Y'), ['4','5','6'],'N'])

Or am I stuck doing a replace for each value?

Thanks!!