How to find distance between 2 lat and 2 long using geospatail using spark scala / pyspark

suppose this is the dataframe 1

Village Tehsil District Type Code POP V_Lat V_Long
Sipahia Tulsipur Balrampur NON Census Village 0 0 27.594705 82.334491
Newazpur Tulsipur Balrampur NON Census Village 0 0 27.605287 82.34746
Ratanpur Tulsipur Balrampur NON Census Village 0 0 27.573511 82.336592
Jumai Dih Tulsipur Balrampur NON Census Village 0 0 27.582564 82.355718
Chhatwapur Tulsipur Balrampur NON Census Village 0 0 27.57687 82.322748
Allah Dih Tulsipur Balrampur NON Census Village 0 0 27.583982 82.344223
Kukarbhukwa Tulsipur Balrampur NON Census Village 0 0 27.577273 82.330141
Udaipur Tulsipur Balrampur NON Census Village 0 0 27.569862 82.326575
Prem Nagar Tulsipur Balrampur Village 173435 2702 27.584897 82.353102
Lal Nagar Tulsipur Balrampur Village 173434 2552 27.592387 82.330867
Khaira Tulsipur Balrampur Village 173436 3506 27.5734 82.340243
Mahadeo Jumuni Tulsipur Balrampur Village 173431 1693 27.599005 82.345086
Bankot Haidergarh Bara Banki NON Census Village 0 0 26.579465 81.461515


and suppose this is second dataframe

UE Purwa Dhanauti Haidergarh Bara Banki NON Census Village 0 0 26.568228 81.471936
UE Purwa Lachhmansingh Haidergarh Bara Banki NON Census Village 0 0 26.569711 81.478505


I need to calculate the distance between these two data frame , only i need to filter out those row that are within 500 meter.

If i simply say if i do the crossjoin two number of rows will be arourd 26 . after that suppose there are only 5 record with in 500 meter i will take only these record.

But the problem is that suppose both data are huge in size than cross join is not good option for joining.

I am looking for the solution in geospatail in spark with scala or spark with pyspark.