Same department. Stay or Go?

I have been at the same department, same branch, same project, since 2017. I am 32 and this is my first ‘real’ job. Before, I was in and out of school trying to figure out life. I was very lucky in my public servant journey. Started off as an AS01 admin, got my indeterminate. AS02 promotion after 1.5 years. PM03 acting for 2years, got my indeterminate. Went on mat leave, and now I am currently acting PM05. I heard it was bad to stay at the same dept for a long time. Given the success I had in the last years (if that even matters) should I stay or look for other opportunities to grow. Does it look on a resume to have worked at only one place?