Small talk in Japanese is less about having something to say and more about signalling that you are listening. A handful of reaction words will carry a conversation further than a large vocabulary.
Coloured text marks the part that matters: usually the word to swap for your own, sometimes the small word doing the work.
Openers: the weather
Weather is the standard neutral opener, and ね is what turns it into a conversation.
今日は暑いですね。
Kyou wa atsui desu ne.
It is hot today, is it not?
Without ね this is a weather report. With ね it invites a reply.
いい天気ですね。
Ii tenki desu ne.
Lovely weather.
寒くなりましたね。
Samuku narimashita ne.
It has got cold, has it not?
Asking about them
お仕事は何ですか。
Oshigoto wa nan desu ka.
What do you do for work?
どちらから来ましたか。
Dochira kara kimashita ka.
Where are you from?
どちら is the polite どこ. Softer for a stranger.
日本は初めてですか。
Nihon wa hajimete desu ka.
Is this your first time in Japan?
週末は何をしましたか。
Shuumatsu wa nani o shimashita ka.
What did you do at the weekend?
Talking about your Japanese
This comes up every single time. Have the answers ready.
二年、日本語を勉強しています。
Ni-nen, nihongo o benkyou shite imasu.
I have been studying Japanese for two years.
少しだけ話せます。
Sukoshi dake hanasemasu.
I can only speak a little.
いえいえ、まだまだです。
Ie ie, mada mada desu.
Not at all, I have a long way to go.
The expected reply to 日本語お上手ですね. Accepting the compliment outright sounds boastful.
漢字が難しいです。
Kanji ga muzukashii desu.
Kanji is hard.
Universally agreed with, including by Japanese people.
Keeping it alive
These are the engine of Japanese conversation. Use them often, they are not filler.
そうなんですか。
Sou nan desu ka.
Oh really?
いいですね。
Ii desu ne.
That is nice.
私もです。
Watashi mo desu.
Me too.
本当ですか。
Hontou desu ka.
Really?
なるほど。
Naruhodo.
I see. / That makes sense.
〜はどうですか。
~ wa dou desu ka.
How about ...?
Bounces the question back and keeps the conversation moving.
Wrapping up
そろそろ失礼します。
Sorosoro shitsurei shimasu.
I should get going.
そろそろ ("soon") softens it. Announcing you are leaving outright is blunt.
楽しかったです。
Tanoshikatta desu.
That was fun.
また会いましょう。
Mata aimashou.
Let us meet again.
Worth knowing
ね is the whole trick
It turns a statement into an invitation to agree. 暑いです is information, 暑いですね is a conversation. Over-use it rather than under-use it.
Reactions matter more than content
Japanese conversation expects frequent aizuchi: そうですか, なるほど, ええ. Listening in silence reads as disengaged, not polite.
Silence is allowed
Pauses are more comfortable in Japanese conversation than in English. You do not have to fill every gap, and rushing to do so can feel pushy.
One-line rule: Open with the weather plus ね, react constantly, deflect the compliment on your Japanese, and use そろそろ to leave without ending things abruptly.